starred/windmill
1
0
Fork 0
mirror of https://github.com/windmill-labs/windmill.git synced 2026-09-10 08:07:03 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'main' into sonner-toast

Browse Source
This commit is contained in:
Faton Ramadani
2024-05-13 13:38:33 +02:00
committed by GitHub
parent 33a38a89b3 79370535f4
commit ff8577b432
154 changed files with 60299 additions and 1466 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/jsr_on_release.yml
+1 -1
View File
@@ -10,7 +10,7 @@ jobs:
runs-on: ubuntu-latest
permissions:
contents: read
id-token: ${{ secrets.JSR_OIDC_ID_TOKEN }}
id-token: write
steps:
- uses: actions/checkout@v4
- run: cd typescript-client && ./publish.jsr.sh
CHANGELOG.md
+84
View File
@@ -1,5 +1,89 @@
# Changelog
## [1.327.0](https://github.com/windmill-labs/windmill/compare/v1.326.1...v1.327.0) (2024-05-13)
### Features
* nativets can use the wmill library + setClient not required anymore ([#3714](https://github.com/windmill-labs/windmill/issues/3714)) ([8b21f08](https://github.com/windmill-labs/windmill/commit/8b21f0812c939542ed84863de029b60f94526417))
## [1.326.1](https://github.com/windmill-labs/windmill/compare/v1.326.0...v1.326.1) (2024-05-12)
### Bug Fixes
* un-inline monaco editor workers for faster initial load ([e2991ef](https://github.com/windmill-labs/windmill/commit/e2991ef9237d17de5e6ef8074387998a96785396))
## [1.326.0](https://github.com/windmill-labs/windmill/compare/v1.325.2...v1.326.0) (2024-05-12)
### Features
* logs can be downloaded directly from server/frontend if using shared volume ([a3a66d0](https://github.com/windmill-labs/windmill/commit/a3a66d0577f57cae6e0de5a942a35d45cc8e9472))
## [1.325.2](https://github.com/windmill-labs/windmill/compare/v1.325.1...v1.325.2) (2024-05-11)
### Bug Fixes
* improve status exit dedicated workers ([12e302a](https://github.com/windmill-labs/windmill/commit/12e302a309ee8148d2a9d4a168807b97f5fd1709))
## [1.325.1](https://github.com/windmill-labs/windmill/compare/v1.325.0...v1.325.1) (2024-05-11)
### Bug Fixes
* **cli:** support whileloop in flow cli sync ([d0a6dda](https://github.com/windmill-labs/windmill/commit/d0a6ddab94dfae4d6fab40c4d41acb85dab18183))
* improve nodejs mode after esm update ([87add79](https://github.com/windmill-labs/windmill/commit/87add79de2e4df38e59d2e334bba0487c8ff4ecc))
* selectedRow for a table is set on table actions ([447e2d7](https://github.com/windmill-labs/windmill/commit/447e2d7742edcd0af7184f6e8e5e67cf99f8103b))
## [1.325.0](https://github.com/windmill-labs/windmill/compare/v1.324.2...v1.325.0) (2024-05-10)
### Features
* worker metrics ([#3697](https://github.com/windmill-labs/windmill/issues/3697)) ([696a561](https://github.com/windmill-labs/windmill/commit/696a5612f8551066b7b833ba555bd006914faff6))
### Bug Fixes
* **backend:** return flow result if flow fails or stops before early return ([#3704](https://github.com/windmill-labs/windmill/issues/3704)) ([67f4a4a](https://github.com/windmill-labs/windmill/commit/67f4a4afeec79daa81930124594ecf3131c18133))
## [1.324.2](https://github.com/windmill-labs/windmill/compare/v1.324.1...v1.324.2) (2024-05-09)
### Bug Fixes
* fix build ([20dfbda](https://github.com/windmill-labs/windmill/commit/20dfbda69e643753b1ff17137ce18ed23ff45dd1))
## [1.324.1](https://github.com/windmill-labs/windmill/compare/v1.324.0...v1.324.1) (2024-05-09)
### Bug Fixes
* **cli:** narrow codebases to bun ([252ac18](https://github.com/windmill-labs/windmill/commit/252ac18d477742d93a2004a0976a809d10bff3d1))
## [1.324.0](https://github.com/windmill-labs/windmill/compare/v1.323.6...v1.324.0) (2024-05-09)
### Features
* critical error side channel ([#3625](https://github.com/windmill-labs/windmill/issues/3625)) ([29b1e6f](https://github.com/windmill-labs/windmill/commit/29b1e6f6a28ac9d9c014e4da2e23c9f2e45e1628))
* disable response logs based on env var ([#3685](https://github.com/windmill-labs/windmill/issues/3685)) ([cd1711c](https://github.com/windmill-labs/windmill/commit/cd1711cf03894e65e81ee74631b0e04d7dae76d2))
* **frontend:** Add a button tocopy the path of a subflow ([#3691](https://github.com/windmill-labs/windmill/issues/3691)) ([630ae7e](https://github.com/windmill-labs/windmill/commit/630ae7ec4b243e41d841bdae41c1c87b0fda5a82))
* **frontend:** add missing date in the previous run panel ([#3693](https://github.com/windmill-labs/windmill/issues/3693)) ([49867c0](https://github.com/windmill-labs/windmill/commit/49867c06e63b5fc7b73031d946effb88c1ad1d1e))
* **frontend:** Improve app components ([#3672](https://github.com/windmill-labs/windmill/issues/3672)) ([ee2a193](https://github.com/windmill-labs/windmill/commit/ee2a193958b87a7557eadfd7ea8231d109b9d41c))
* import export all worker groups config ([#3667](https://github.com/windmill-labs/windmill/issues/3667)) ([c1a4a82](https://github.com/windmill-labs/windmill/commit/c1a4a8284387f381a5882b1fb6ebd73c9ec6b1ba))
* local typescript codebase as bundle ([#3694](https://github.com/windmill-labs/windmill/issues/3694)) ([11b3ea3](https://github.com/windmill-labs/windmill/commit/11b3ea3ac86a34ab1beaab98ea43f6a64879f0fb))
### Bug Fixes
* can cancel only if can disable schedule + stop worker on force cancel + soft cancel job parent on job cancel ([#3670](https://github.com/windmill-labs/windmill/issues/3670)) ([010662d](https://github.com/windmill-labs/windmill/commit/010662d3d0d05a7f6167ceeb592c97cec4c9e804))
* **frontend:** Fix subflow viewer ([#3674](https://github.com/windmill-labs/windmill/issues/3674)) ([47ee6c9](https://github.com/windmill-labs/windmill/commit/47ee6c90998840889c44df28a25967cef6c8bd59))
* improve conditional wrapper for app editor ([ea4165d](https://github.com/windmill-labs/windmill/commit/ea4165d58bb11fb4409242bb597f4b7514d1bd60))
* load large args of previous runs dynamically ([#3688](https://github.com/windmill-labs/windmill/issues/3688)) ([d1f58f1](https://github.com/windmill-labs/windmill/commit/d1f58f10371dfce5e037f03b43aad9f530d7ae6a))
## [1.323.6](https://github.com/windmill-labs/windmill/compare/v1.323.5...v1.323.6) (2024-05-02)
Dockerfile
+1 -1
View File
@@ -171,7 +171,7 @@ COPY --from=builder /windmill/target/release/windmill ${APP}/windmill
COPY --from=downloader --chmod=755 /deno /usr/bin/deno
COPY --from=oven/bun:1.1.5 /usr/local/bin/bun /usr/bin/bun
COPY --from=oven/bun:1.1.7 /usr/local/bin/bun /usr/bin/bun
# add the docker client to call docker from a worker if enabled
COPY --from=docker:dind /usr/local/bin/docker /usr/local/bin/
README.md
+1
View File
@@ -352,6 +352,7 @@ you to have it being synced automatically everyday.
| TIMEOUT_WAIT_RESULT | 20 | The number of seconds to wait before timeout on the 'run_wait_result' endpoint | Worker |
| QUEUE_LIMIT_WAIT_RESULT | None | The number of max jobs in the queue before rejecting immediately the request in 'run_wait_result' endpoint. Takes precedence on the query arg. If none is specified, there are no limit. | Worker |
| DENO_AUTH_TOKENS | None | Custom DENO_AUTH_TOKENS to pass to worker to allow the use of private modules | Worker |
| DISABLE_RESPONSE_LOGS | false | Disable response logs | Server |
## Run a local dev setup
### only Frontend
backend/.sqlx/query-021be0f26ea87e587e656b24a9a94538efbf54a1447a3898e19773789cfc9063.json
Generated
+20
View File
@@ -0,0 +1,20 @@
{
"db_name": "PostgreSQL",
"query": "SELECT created_at FROM metrics WHERE id = 'author_count' ORDER BY created_at DESC LIMIT 1",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "created_at",
"type_info": "Timestamptz"
}
],
"parameters": {
"Left": []
},
"nullable": [
false
]
},
"hash": "021be0f26ea87e587e656b24a9a94538efbf54a1447a3898e19773789cfc9063"
}
backend/.sqlx/query-3e0e5e0076ae0f7771abd64f8c5bb7e003e3f85351c946a012037e0c874acf1a.json → backend/.sqlx/query-02b516dac764662194db1bc33e365c01f40bae70af3683f1f09748f6020f0d49.json
Generated
+2 -2
View File
@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "SELECT tag, count(*) as count FROM queue WHERE\n scheduled_for <= now() - ('3 seconds')::interval AND running = false\n GROUP BY tag",
"query": "SELECT tag, count(*) as count FROM queue WHERE\n scheduled_for <= now() - ('3 seconds')::interval AND running = false\n GROUP BY tag",
"describe": {
"columns": [
{
@@ -22,5 +22,5 @@
null
]
},
"hash": "3e0e5e0076ae0f7771abd64f8c5bb7e003e3f85351c946a012037e0c874acf1a"
"hash": "02b516dac764662194db1bc33e365c01f40bae70af3683f1f09748f6020f0d49"
}
backend/.sqlx/query-3667d72d23e8c35ab96d5e5d1fbfd94a8a9f74b6398e21f13c206c3f9427c6a9.json
Generated
+12
View File
@@ -0,0 +1,12 @@
{
"db_name": "PostgreSQL",
"query": "DELETE FROM metrics \n WHERE (id = 'author_count' OR id = 'operator_count' OR id = 'worker_usage') AND created_at < NOW() - INTERVAL '6 month'",
"describe": {
"columns": [],
"parameters": {
"Left": []
},
"nullable": []
},
"hash": "3667d72d23e8c35ab96d5e5d1fbfd94a8a9f74b6398e21f13c206c3f9427c6a9"
}
backend/.sqlx/query-4887db074e9058bc0c1887428d6b2a897fe4b30c83f2aa5e9464c156f08f1c3a.json
Generated
+38
View File
@@ -0,0 +1,38 @@
{
"db_name": "PostgreSQL",
"query": "SELECT worker, worker_instance, vcpus, memory FROM worker_ping WHERE ping_at > NOW() - INTERVAL '2 minutes'",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "worker",
"type_info": "Varchar"
},
{
"ordinal": 1,
"name": "worker_instance",
"type_info": "Varchar"
},
{
"ordinal": 2,
"name": "vcpus",
"type_info": "Int8"
},
{
"ordinal": 3,
"name": "memory",
"type_info": "Int8"
}
],
"parameters": {
"Left": []
},
"nullable": [
false,
false,
true,
true
]
},
"hash": "4887db074e9058bc0c1887428d6b2a897fe4b30c83f2aa5e9464c156f08f1c3a"
}
backend/.sqlx/query-47beea5cd6324b53bfb349665fb215280f32b70a617fde87f70ea53ca9ade39f.json → backend/.sqlx/query-54fef88cc6b9e8db7c07fccbaa845edfefa339153a32e468faad9f063008863d.json
Generated
+3 -2
View File
@@ -1,16 +1,17 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE worker_ping SET ping_at = now(), jobs_executed = $1, custom_tags = $2 WHERE worker = $3",
"query": "UPDATE worker_ping SET ping_at = now(), jobs_executed = $1, custom_tags = $2, occupancy_rate = $3, current_job_id = NULL, current_job_workspace_id = NULL WHERE worker = $4",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Int4",
"TextArray",
"Float4",
"Text"
]
},
"nullable": []
},
"hash": "47beea5cd6324b53bfb349665fb215280f32b70a617fde87f70ea53ca9ade39f"
"hash": "54fef88cc6b9e8db7c07fccbaa845edfefa339153a32e468faad9f063008863d"
}
backend/.sqlx/query-5ce9a9e0669b299b998bc9080d24f8d6f56eb049bec2e0745d27ca3a4fc0c296.json
Generated
+14
View File
@@ -0,0 +1,14 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO metrics (id, value) VALUES ('worker_usage', $1)",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Jsonb"
]
},
"nullable": []
},
"hash": "5ce9a9e0669b299b998bc9080d24f8d6f56eb049bec2e0745d27ca3a4fc0c296"
}
backend/.sqlx/query-661f472ff3860983322162420457f5033b9c9afc344d9c3e385ba20a3ad2197a.json
Generated
+1 -1
View File
@@ -5,7 +5,7 @@
"columns": [
{
"ordinal": 0,
"name": "?column?",
"name": "bool",
"type_info": "Bool"
}
],
backend/.sqlx/query-6afc5c7cbb3abe11ade0cedf1f7328005ce4de3165cdd998e5a0d27e044c7153.json
Generated
+22
View File
@@ -0,0 +1,22 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO worker_ping (worker_instance, worker, ip, custom_tags, worker_group, dedicated_worker, wm_version, vcpus, memory) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9) ON CONFLICT (worker) DO UPDATE set ip = $3, custom_tags = $4, worker_group = $5",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"Varchar",
"Varchar",
"TextArray",
"Varchar",
"Varchar",
"Varchar",
"Int8",
"Int8"
]
},
"nullable": []
},
"hash": "6afc5c7cbb3abe11ade0cedf1f7328005ce4de3165cdd998e5a0d27e044c7153"
}
backend/.sqlx/query-7b609ed87f974dfe887778746230b40b485658a75a1186c6ebb4c2de1b52a2fa.json
Generated
-20
View File
@@ -1,20 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO worker_ping (worker_instance, worker, ip, custom_tags, worker_group, dedicated_worker, wm_version) VALUES ($1, $2, $3, $4, $5, $6, $7) ON CONFLICT (worker) DO UPDATE set ip = $3, custom_tags = $4, worker_group = $5",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"Varchar",
"Varchar",
"TextArray",
"Varchar",
"Varchar",
"Varchar"
]
},
"nullable": []
},
"hash": "7b609ed87f974dfe887778746230b40b485658a75a1186c6ebb4c2de1b52a2fa"
}
backend/.sqlx/query-7c95f3652de4561b18c1a340db3e0113a4777d38de4468fc41b22ce5b6fd1db7.json
Generated
+26
View File
@@ -0,0 +1,26 @@
{
"db_name": "PostgreSQL",
"query": "WITH queue_metrics as (\n SELECT id, value, created_at\n FROM metrics\n WHERE id LIKE 'queue_%'\n AND created_at > now() - interval '14 day'\n ORDER BY created_at ASC\n )\n SELECT id, array_agg(json_build_object('value', value, 'created_at', created_at)) as \"values!\"\n FROM queue_metrics\n GROUP BY id\n ORDER BY id ASC",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "id",
"type_info": "Varchar"
},
{
"ordinal": 1,
"name": "values!",
"type_info": "JsonArray"
}
],
"parameters": {
"Left": []
},
"nullable": [
false,
null
]
},
"hash": "7c95f3652de4561b18c1a340db3e0113a4777d38de4468fc41b22ce5b6fd1db7"
}
backend/.sqlx/query-7c9a464ac807051b99fe37f2078f1b17f824e6d9b1124db618855a15a98e31f6.json
Generated
+12
View File
@@ -0,0 +1,12 @@
{
"db_name": "PostgreSQL",
"query": "DELETE FROM metrics WHERE id LIKE 'queue_%' AND created_at < NOW() - INTERVAL '14 day'",
"describe": {
"columns": [],
"parameters": {
"Left": []
},
"nullable": []
},
"hash": "7c9a464ac807051b99fe37f2078f1b17f824e6d9b1124db618855a15a98e31f6"
}
backend/.sqlx/query-82f6674f19e8ad51a992505a46f46fc4a48172f104e9e849f755ac041c3eef92.json
Generated
+20
View File
@@ -0,0 +1,20 @@
{
"db_name": "PostgreSQL",
"query": "SELECT created_at FROM metrics WHERE id LIKE 'queue_count_%' ORDER BY created_at DESC LIMIT 1",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "created_at",
"type_info": "Timestamptz"
}
],
"parameters": {
"Left": []
},
"nullable": [
false
]
},
"hash": "82f6674f19e8ad51a992505a46f46fc4a48172f104e9e849f755ac041c3eef92"
}
backend/.sqlx/query-8824b382c4e98dfa17b4aa656af3a6c1ff99973e778d71bd598a50d022da8f15.json
Generated
+15
View File
@@ -0,0 +1,15 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO metrics (id, value) VALUES ($1, $2)",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"Jsonb"
]
},
"nullable": []
},
"hash": "8824b382c4e98dfa17b4aa656af3a6c1ff99973e778d71bd598a50d022da8f15"
}
backend/.sqlx/query-8c62e3bb264c7336b3b3a68677993ba3b729c7358cc3da14513d8b447ce65aaf.json
Generated
+16
View File
@@ -0,0 +1,16 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE worker_ping SET ping_at = now(), current_job_id = $1, current_job_workspace_id = $2 WHERE worker = $3",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Uuid",
"Varchar",
"Text"
]
},
"nullable": []
},
"hash": "8c62e3bb264c7336b3b3a68677993ba3b729c7358cc3da14513d8b447ce65aaf"
}
backend/.sqlx/query-92d00c6a1f4c40f2a23c9ba758a59597deabd7ca93653ad72d2cdc37efefc9d4.json → backend/.sqlx/query-8e7ff45c5378c3a3406ba94dc0653afa5d28c072203617c598caefaaf1bafcfb.json
Generated
+4 -3
View File
@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO script (workspace_id, hash, path, parent_hashes, summary, description, content, created_by, schema, is_template, extra_perms, lock, language, kind, tag, draft_only, envs, concurrent_limit, concurrency_time_window_s, cache_ttl, dedicated_worker, ws_error_handler_muted, priority, restart_unless_cancelled, delete_after_use, timeout, concurrency_key, visible_to_runner_only, no_main_func) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9::text::json, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22, $23, $24, $25, $26, $27, $28, $29)",
"query": "INSERT INTO script (workspace_id, hash, path, parent_hashes, summary, description, content, created_by, schema, is_template, extra_perms, lock, language, kind, tag, draft_only, envs, concurrent_limit, concurrency_time_window_s, cache_ttl, dedicated_worker, ws_error_handler_muted, priority, restart_unless_cancelled, delete_after_use, timeout, concurrency_key, visible_to_runner_only, no_main_func, codebase) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9::text::json, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22, $23, $24, $25, $26, $27, $28, $29, $30)",
"describe": {
"columns": [],
"parameters": {
@@ -67,10 +67,11 @@
"Int4",
"Varchar",
"Bool",
"Bool"
"Bool",
"Varchar"
]
},
"nullable": []
},
"hash": "92d00c6a1f4c40f2a23c9ba758a59597deabd7ca93653ad72d2cdc37efefc9d4"
"hash": "8e7ff45c5378c3a3406ba94dc0653afa5d28c072203617c598caefaaf1bafcfb"
}
backend/.sqlx/query-9bf41c3161a02b7d0731c4e1d79519cef5255f5df1b759af3aa4985bb64313e5.json
Generated
+15
View File
@@ -0,0 +1,15 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO metrics (id, value)\n VALUES ($1, to_jsonb((SELECT EXTRACT(EPOCH FROM now() - scheduled_for)\n FROM queue WHERE tag = $2 AND running = false AND scheduled_for <= now() - ('3 seconds')::interval\n ORDER BY priority DESC NULLS LAST, scheduled_for, created_at LIMIT 1)))",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"Text"
]
},
"nullable": []
},
"hash": "9bf41c3161a02b7d0731c4e1d79519cef5255f5df1b759af3aa4985bb64313e5"
}
backend/.sqlx/query-bbd5f968d7b62a55a7ebf7b98cfd411678ce544fdb9472d8d223235fb2818aaf.json
Generated
+23
View File
@@ -0,0 +1,23 @@
{
"db_name": "PostgreSQL",
"query": "SELECT args FROM completed_job WHERE id = $1 AND workspace_id = $2 UNION ALL SELECT args FROM input WHERE id = $1 AND workspace_id = $2",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "args",
"type_info": "Jsonb"
}
],
"parameters": {
"Left": [
"Uuid",
"Text"
]
},
"nullable": [
null
]
},
"hash": "bbd5f968d7b62a55a7ebf7b98cfd411678ce544fdb9472d8d223235fb2818aaf"
}
backend/.sqlx/query-c584aeec21716f7405c355d5684ff3ac1be3f70849d67b8c2d4787e809b7ea2b.json
Generated
+26
View File
@@ -0,0 +1,26 @@
{
"db_name": "PostgreSQL",
"query": "WITH all_users as (SELECT count(*)::INT as count FROM usr WHERE disabled IS false),\n authors as (SELECT count(distinct email)::INT as count FROM usr WHERE usr.operator IS false AND disabled IS false)\n SELECT authors.count as author_count, all_users.count - authors.count as operator_count FROM all_users, authors",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "author_count",
"type_info": "Int4"
},
{
"ordinal": 1,
"name": "operator_count",
"type_info": "Int4"
}
],
"parameters": {
"Left": []
},
"nullable": [
null,
null
]
},
"hash": "c584aeec21716f7405c355d5684ff3ac1be3f70849d67b8c2d4787e809b7ea2b"
}
backend/.sqlx/query-c6b2791dd109c7bf40a40b557d9d5a140e70c49718fcb3a776b77f8f2ab901d4.json
Generated
+15
View File
@@ -0,0 +1,15 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO metrics (id, value) VALUES ('author_count', $1), ('operator_count', $2)",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Jsonb",
"Jsonb"
]
},
"nullable": []
},
"hash": "c6b2791dd109c7bf40a40b557d9d5a140e70c49718fcb3a776b77f8f2ab901d4"
}
backend/.sqlx/query-c9d97800eb0ec87df8e8959b283dacb2c6cce422365ed394375641488ceb6b65.json
Generated
-14
View File
@@ -1,14 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE worker_ping SET ping_at = now() WHERE worker = $1",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text"
]
},
"nullable": []
},
"hash": "c9d97800eb0ec87df8e8959b283dacb2c6cce422365ed394375641488ceb6b65"
}
backend/.sqlx/query-b38044d94e2ab03167c2f6fbb553ab3c19930ed11abf51763cd3ee378229443d.json → backend/.sqlx/query-e00171cc3fc8f32922562d4fc4d3db56955fbc6ee9872d4b2e99157ebed131e0.json
Generated
+26 -7
View File
@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "SELECT worker, worker_instance, EXTRACT(EPOCH FROM (now() - ping_at))::integer as last_ping, started_at, ip, jobs_executed, custom_tags, worker_group, wm_version FROM worker_ping\n WHERE ($1::integer IS NULL AND ping_at > now() - interval '5 minute') OR (ping_at > now() - ($1 || ' seconds')::interval)\n ORDER BY ping_at desc LIMIT $2 OFFSET $3",
"query": "SELECT worker, worker_instance, EXTRACT(EPOCH FROM (now() - ping_at))::integer as last_ping, started_at, ip, jobs_executed, CASE WHEN $4 IS TRUE THEN current_job_id ELSE NULL END as current_job_id, CASE WHEN $4 IS TRUE THEN current_job_workspace_id ELSE NULL END as current_job_workspace_id, custom_tags, worker_group, wm_version, occupancy_rate\n FROM worker_ping\n WHERE ($1::integer IS NULL AND ping_at > now() - interval '5 minute') OR (ping_at > now() - ($1 || ' seconds')::interval)\n ORDER BY ping_at desc LIMIT $2 OFFSET $3",
"describe": {
"columns": [
{
@@ -35,25 +35,41 @@
},
{
"ordinal": 6,
"name": "custom_tags",
"type_info": "TextArray"
"name": "current_job_id",
"type_info": "Uuid"
},
{
"ordinal": 7,
"name": "worker_group",
"name": "current_job_workspace_id",
"type_info": "Varchar"
},
{
"ordinal": 8,
"name": "custom_tags",
"type_info": "TextArray"
},
{
"ordinal": 9,
"name": "worker_group",
"type_info": "Varchar"
},
{
"ordinal": 10,
"name": "wm_version",
"type_info": "Varchar"
},
{
"ordinal": 11,
"name": "occupancy_rate",
"type_info": "Float4"
}
],
"parameters": {
"Left": [
"Int4",
"Int8",
"Int8"
"Int8",
"Bool"
]
},
"nullable": [
@@ -63,10 +79,13 @@
false,
false,
false,
null,
null,
true,
false,
false
false,
true
]
},
"hash": "b38044d94e2ab03167c2f6fbb553ab3c19930ed11abf51763cd3ee378229443d"
"hash": "e00171cc3fc8f32922562d4fc4d3db56955fbc6ee9872d4b2e99157ebed131e0"
}
backend/Cargo.lock
Generated
+241 -217
View File
File diff suppressed because it is too large Load Diff
backend/Cargo.toml
+3 -3
View File
@@ -1,6 +1,6 @@
[package]
name = "windmill"
version = "1.323.6"
version = "1.327.0"
authors.workspace = true
edition.workspace = true
@@ -24,7 +24,7 @@ members = [
]
[workspace.package]
version = "1.323.6"
version = "1.327.0"
authors = ["Ruben Fiszel <ruben@windmill.dev>"]
edition = "2021"
@@ -109,7 +109,7 @@ windmill-parser-sql = { path = "./parsers/windmill-parser-sql" }
windmill-parser-graphql = { path = "./parsers/windmill-parser-graphql" }
windmill-api-client = { path = "./windmill-api-client" }
axum = { version = "^0.7" }
axum = { version = "^0.7", features = ["multipart"] }
headers = "^0"
hyper = { version = "^1", features = ["full"] }
tokio = { version = "^1", features = ["full", "tracing"] }
backend/ee-repo-ref.txt
+1 -1
View File
@@ -1 +1 @@
f59f9c1e55e5e93f9eb6c081847cc566611029d2
644b6f49f087790a728a7a0a82525997b1742738
backend/migrations/20240430005030_add_codebases.down.sql
+2
View File
@@ -0,0 +1,2 @@
-- Add down migration script here
ALTER TABLE script DROP COLUMN codebase;
backend/migrations/20240430005030_add_codebases.up.sql
+3
View File
@@ -0,0 +1,3 @@
-- Add up migration script here
ALTER TABLE script ADD COLUMN codebase VARCHAR(255);
backend/migrations/20240506085935_add_worker_ping_metrics.down.sql
+1
View File
@@ -0,0 +1 @@
-- Add down migration script here
backend/migrations/20240506085935_add_worker_ping_metrics.up.sql
+7
View File
@@ -0,0 +1,7 @@
-- Add up migration script here
ALTER TABLE worker_ping
ADD COLUMN current_job_id UUID,
ADD COLUMN current_job_workspace_id VARCHAR(50),
ADD COLUMN vcpus BIGINT,
ADD COLUMN memory BIGINT,
ADD COLUMN occupancy_rate REAL;
backend/migrations/20240506114033_add_metrics_table.down.sql
+1
View File
@@ -0,0 +1 @@
-- Add down migration script here
backend/migrations/20240506114033_add_metrics_table.up.sql
+8
View File
@@ -0,0 +1,8 @@
-- Add up migration script here
CREATE TABLE metrics (
id VARCHAR(255) NOT NULL,
value JSONB NOT NULL,
created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP NOT NULL
);
CREATE INDEX metrics_key_idx ON metrics(id);
CREATE INDEX metrics_sort_idx ON metrics(created_at DESC);
backend/src/monitor.rs
+162 -19
View File
@@ -8,7 +8,7 @@ use std::{
};
use rsmq_async::MultiplexedRsmq;
use serde::de::DeserializeOwned;
use serde::{de::DeserializeOwned, Serialize};
use sqlx::{Pool, Postgres};
use tokio::{
join,
@@ -678,39 +678,182 @@ pub async fn monitor_db(
};
let expose_queue_metrics_f = async {
if !initial_load
&& METRICS_ENABLED.load(std::sync::atomic::Ordering::Relaxed)
&& server_mode
{
if !initial_load && server_mode {
expose_queue_metrics(&db).await;
}
};
let save_usage_metrics_f = async {
if !initial_load && server_mode {
save_usage_metrics(&db).await;
}
};
join!(
expired_items_f,
zombie_jobs_f,
expose_queue_metrics_f,
save_usage_metrics_f,
verify_license_key_f
);
}
pub async fn expose_queue_metrics(db: &Pool<Postgres>) {
let queue_counts = sqlx::query!(
"SELECT tag, count(*) as count FROM queue WHERE
scheduled_for <= now() - ('3 seconds')::interval AND running = false
GROUP BY tag"
)
.fetch_all(db)
.await
.ok()
.unwrap_or_else(|| vec![]);
for q in queue_counts {
let count = q.count.unwrap_or(0);
let tag = q.tag;
let metric = (*QUEUE_COUNT).with_label_values(&[&tag]);
metric.set(count as i64);
let tx = db.begin().await;
if let Ok(mut tx) = tx {
let last_check = sqlx::query_scalar!(
"SELECT created_at FROM metrics WHERE id LIKE 'queue_count_%' ORDER BY created_at DESC LIMIT 1"
)
.fetch_optional(db)
.await
.unwrap_or(Some(chrono::Utc::now()));
let metrics_enabled = METRICS_ENABLED.load(std::sync::atomic::Ordering::Relaxed);
let save_metrics = last_check
.map(|last_check| chrono::Utc::now() - last_check > chrono::Duration::seconds(25))
.unwrap_or(true);
if metrics_enabled || save_metrics {
let queue_counts = sqlx::query!(
"SELECT tag, count(*) as count FROM queue WHERE
scheduled_for <= now() - ('3 seconds')::interval AND running = false
GROUP BY tag"
)
.fetch_all(&mut *tx)
.await
.ok()
.unwrap_or_else(|| vec![]);
for q in queue_counts {
let count = q.count.unwrap_or(0);
let tag = q.tag;
if metrics_enabled {
let metric = (*QUEUE_COUNT).with_label_values(&[&tag]);
metric.set(count as i64);
}
// save queue_count and delay metrics per tag
if save_metrics {
sqlx::query!(
"INSERT INTO metrics (id, value) VALUES ($1, $2)",
format!("queue_count_{}", tag),
serde_json::json!(count)
)
.execute(&mut *tx)
.await
.ok();
if count > 0 {
sqlx::query!(
"INSERT INTO metrics (id, value)
VALUES ($1, to_jsonb((SELECT EXTRACT(EPOCH FROM now() - scheduled_for)
FROM queue WHERE tag = $2 AND running = false AND scheduled_for <= now() - ('3 seconds')::interval
ORDER BY priority DESC NULLS LAST, scheduled_for, created_at LIMIT 1)))",
format!("queue_delay_{}", tag),
tag
).execute(&mut *tx).await.ok();
}
}
}
}
// clean queue metrics older than 14 days
sqlx::query!(
"DELETE FROM metrics WHERE id LIKE 'queue_%' AND created_at < NOW() - INTERVAL '14 day'"
).execute(&mut *tx).await.ok();
tx.commit().await.ok();
}
}
#[derive(Serialize)]
struct WorkerUsage {
worker: String,
worker_instance: String,
vcpus: Option<i64>,
memory: Option<i64>,
}
pub async fn save_usage_metrics(db: &Pool<Postgres>) {
let tx = db.begin().await;
if let Ok(mut tx) = tx {
let last_check = sqlx::query_scalar!(
"SELECT created_at FROM metrics WHERE id = 'author_count' ORDER BY created_at DESC LIMIT 1"
)
.fetch_optional(db)
.await
.unwrap_or(Some(chrono::Utc::now()));
let random_nb = rand::random::<i64>();
// save author and operator count every ~24 hours
if last_check
.map(|last_check| chrono::Utc::now() - last_check > chrono::Duration::hours(24) - chrono::Duration::minutes(random_nb % 60))
.unwrap_or(true)
{
let counts = sqlx::query!(
"WITH all_users as (SELECT count(*)::INT as count FROM usr WHERE disabled IS false),
authors as (SELECT count(distinct email)::INT as count FROM usr WHERE usr.operator IS false AND disabled IS false)
SELECT authors.count as author_count, all_users.count - authors.count as operator_count FROM all_users, authors"
)
.fetch_one(&mut *tx)
.await
.ok();
if let Some(counts) = counts {
sqlx::query!(
"INSERT INTO metrics (id, value) VALUES ('author_count', $1), ('operator_count', $2)",
serde_json::json!(counts.author_count),
serde_json::json!(counts.operator_count)
)
.execute(&mut *tx)
.await
.ok();
}
// clean metrics older than 6 months (including worker usage)
sqlx::query!(
"DELETE FROM metrics
WHERE (id = 'author_count' OR id = 'operator_count' OR id = 'worker_usage') AND created_at < NOW() - INTERVAL '6 month'"
)
.execute(&mut *tx)
.await
.ok();
}
// save worker usage every ~60 minutes
if last_check
.map(|last_check| chrono::Utc::now() - last_check > chrono::Duration::minutes(60) - chrono::Duration::seconds(random_nb % 300))
.unwrap_or(true)
{
let worker_usage = sqlx::query_as!(
WorkerUsage,
"SELECT worker, worker_instance, vcpus, memory FROM worker_ping WHERE ping_at > NOW() - INTERVAL '2 minutes'"
)
.fetch_all(&mut *tx)
.await
.ok();
if let Some(worker_usage) = worker_usage {
sqlx::query!(
"INSERT INTO metrics (id, value) VALUES ('worker_usage', $1)",
serde_json::json!(worker_usage)
)
.execute(&mut *tx)
.await
.ok();
}
}
tx.commit().await.ok();
}
}
pub async fn reload_server_config(db: &Pool<Postgres>) {
let config = load_server_config(&db).await;
if let Err(e) = config {
backend/tests/worker.rs
+7
View File
@@ -1663,6 +1663,7 @@ func main(derp string) (string, error) {
.to_owned();
let result = RunJob::from(JobPayload::Code(RawCode {
hash: None,
content,
path: None,
lock: None,
@@ -1694,6 +1695,7 @@ echo "hello $msg"
.to_owned();
let job = RunJob::from(JobPayload::Code(RawCode {
hash: None,
content,
path: None,
lock: None,
@@ -1722,6 +1724,7 @@ def main():
.to_owned();
let job = JobPayload::Code(RawCode {
hash: None,
content,
path: None,
language: ScriptLang::Python3,
@@ -1756,6 +1759,7 @@ def main():
.to_owned();
let job = JobPayload::Code(RawCode {
hash: None,
content,
path: None,
language: ScriptLang::Python3,
@@ -1789,6 +1793,7 @@ def main():
.to_owned();
let job = JobPayload::Code(RawCode {
hash: None,
content,
path: None,
language: ScriptLang::Python3,
@@ -3165,6 +3170,7 @@ async fn run_deployed_relative_imports(db: &Pool<Postgres>, script_content: Stri
concurrency_key: None,
visible_to_runner_only: None,
no_main_func: None,
codebase: None
},
).await.unwrap();
@@ -3212,6 +3218,7 @@ async fn run_preview_relative_imports(db: &Pool<Postgres>, script_content: Strin
let db2 = db.clone();
in_test_worker(&db, async move {
let job = RunJob::from(JobPayload::Code(RawCode {
hash: None,
content: script_content,
path: Some("f/system/test_import".to_string()),
language,
backend/windmill-api/openapi.yaml
+67 -1
View File
@@ -1,7 +1,7 @@
openapi: "3.0.3"
info:
version: 1.323.6
version: 1.327.0
title: Windmill API
contact:
@@ -6959,6 +6959,40 @@ paths:
schema:
type: boolean
/workers/queue_metrics:
get:
summary: get queue metrics
operationId: getQueueMetrics
tags:
- worker
responses:
"200":
description: metrics
content:
application/json:
schema:
type: array
items:
type: object
properties:
id:
type: string
values:
type: array
items:
type: object
properties:
created_at:
type: string
value:
type: number
required:
- created_at
- value
required:
- id
- values
/configs/list_worker_groups:
get:
summary: list worker groups
@@ -7271,6 +7305,26 @@ paths:
items:
$ref: "#/components/schemas/Input"
/w/{workspace}/inputs/{jobOrInputId}/args:
get:
summary: Get args from history or saved input
operationId: getArgsFromHistoryOrSavedInput
tags:
- input
parameters:
- $ref: "#/components/parameters/WorkspaceId"
- name: jobOrInputId
in: path
required: true
schema:
type: string
responses:
"200":
description: args
content:
application/json:
schema: {}
/w/{workspace}/inputs/list:
get:
summary: List saved Inputs for a Runnable
@@ -8390,6 +8444,9 @@ components:
type: boolean
no_main_func:
type: boolean
codebase:
type: string
required:
- hash
- path
@@ -8407,6 +8464,7 @@ components:
- starred
- no_main_func
NewScript:
type: object
properties:
@@ -8481,6 +8539,8 @@ components:
type: boolean
no_main_func:
type: boolean
codebase:
type: string
required:
- path
- summary
@@ -9678,6 +9738,12 @@ components:
type: string
wm_version:
type: string
current_job_id:
type: string
current_job_workspace_id:
type: string
occupancy_rate:
type: number
required:
- worker
- worker_instance
backend/windmill-api/src/inputs.rs
+28 -3
View File
@@ -25,7 +25,7 @@ use windmill_common::{
error::JsonResult,
jobs::JobKind,
scripts::to_i64,
utils::{paginate, Pagination},
utils::{not_found_if_none, paginate, Pagination},
};
pub fn workspaced_service() -> Router {
Router::new()
@@ -34,6 +34,10 @@ pub fn workspaced_service() -> Router {
.route("/create", post(create_input))
.route("/update", post(update_input))
.route("/delete/:id", post(delete_input))
.route(
"/:job_or_input_id/args",
get(get_args_from_history_or_saved_input),
)
}
#[derive(Debug, sqlx::FromRow, Serialize, Deserialize)]
@@ -123,7 +127,7 @@ async fn get_input_history(
let mut tx = user_db.begin(&authed).await?;
let sql = &format!(
"select id, created_at, created_by, args, success from completed_job \
"select id, created_at, created_by, CASE WHEN args is null or pg_column_size(args) < 40000 THEN args ELSE '\"WINDMILL_TOO_BIG\"'::jsonb END as args, success from completed_job \
where {} = $1 and job_kind = $2 and workspace_id = $3 \
order by created_at desc limit $4 offset $5",
r.runnable_type.column_name()
@@ -167,6 +171,27 @@ async fn get_input_history(
Ok(Json(inputs))
}
async fn get_args_from_history_or_saved_input(
authed: ApiAuthed,
Extension(user_db): Extension<UserDB>,
Path((w_id, job_or_input_id)): Path<(String, Uuid)>,
) -> JsonResult<Option<Value>> {
let mut tx = user_db.begin(&authed).await?;
let result_o = sqlx::query_scalar!(
"SELECT args FROM completed_job WHERE id = $1 AND workspace_id = $2 UNION ALL SELECT args FROM input WHERE id = $1 AND workspace_id = $2",
job_or_input_id,
w_id
)
.fetch_optional(&mut *tx)
.await?;
tx.commit().await?;
let result = not_found_if_none(result_o, "Input args", job_or_input_id.to_string())?;
Ok(Json(result))
}
async fn list_saved_inputs(
authed: ApiAuthed,
Extension(user_db): Extension<UserDB>,
@@ -179,7 +204,7 @@ async fn list_saved_inputs(
let mut tx = user_db.begin(&authed).await?;
let rows = sqlx::query_as::<_, InputRow>(
"select * from input \
"select id, workspace_id, runnable_id, runnable_type, name, CASE WHEN pg_column_size(args) < 40000 THEN args ELSE '\"WINDMILL_TOO_BIG\"'::jsonb END as args, created_at, created_by, is_public from input \
where runnable_id = $1 and runnable_type = $2 and workspace_id = $3 \
and (is_public IS true OR created_by = $4) \
order by created_at desc limit $5 offset $6",
backend/windmill-api/src/jobs.rs
+204 -51
View File
@@ -12,6 +12,7 @@ use serde_json::value::RawValue;
use std::collections::HashMap;
#[cfg(feature = "prometheus")]
use std::sync::atomic::Ordering;
use tokio::io::AsyncReadExt;
#[cfg(feature = "prometheus")]
use tokio::time::Instant;
use windmill_common::flow_status::{JobResult, RestartedFrom};
@@ -19,6 +20,10 @@ use windmill_common::jobs::{
format_completed_job_result, format_result, CompletedJobWithFormattedResult, FormattedResult,
ENTRYPOINT_OVERRIDE,
};
use windmill_common::worker::TMP_DIR;
#[cfg(all(feature = "enterprise", feature = "parquet"))]
use windmill_common::scripts::PREVIEW_IS_CODEBASE_HASH;
use windmill_common::variables::get_workspace_key;
use crate::db::ApiAuthed;
@@ -163,6 +168,7 @@ pub fn workspaced_service() -> Router {
.layer(cors.clone()),
)
.route("/run/preview", post(run_preview_script))
.route("/run/preview_bundle", post(run_bundle_preview_script))
.route("/add_batch_jobs/:n", post(add_batch_jobs))
.route("/run/preview_flow", post(run_preview_flow_job))
.route(
@@ -660,17 +666,17 @@ async fn get_job_internal(
async fn get_logs_from_store(
log_offset: i32,
logs: &str,
log_file_index: Option<Vec<String>>,
log_file_index: &Option<Vec<String>>,
) -> Option<error::Result<Body>> {
if log_offset > 0 {
if let Some(file_index) = log_file_index {
if let Some(file_index) = log_file_index.clone() {
tracing::debug!("Getting logs from store: {file_index:?}");
if let Some(os) = OBJECT_STORE_CACHE_SETTINGS.read().await.clone() {
tracing::debug!("object store client present, streaming from there");
let logs = logs.to_string();
let stream = async_stream::stream! {
for file_p in file_index {
for file_p in file_index.clone() {
let file_p_2 = file_p.clone();
let file = os.get(&object_store::path::Path::from(file_p)).await;
if let Ok(file) = file {
@@ -692,6 +698,40 @@ async fn get_logs_from_store(
}
return None;
}
async fn get_logs_from_disk(
log_offset: i32,
logs: &str,
log_file_index: &Option<Vec<String>>,
) -> Option<error::Result<Body>> {
if log_offset > 0 {
if let Some(file_index) = log_file_index.clone() {
for file_p in &file_index {
if !tokio::fs::metadata(format!("{TMP_DIR}/{file_p}"))
.await
.is_ok()
{
return None;
}
}
let logs = logs.to_string();
let stream = async_stream::stream! {
for file_p in file_index.clone() {
let mut file = tokio::fs::File::open(format!("{TMP_DIR}/{file_p}")).await.map_err(to_anyhow)?;
let mut buffer = Vec::new();
file.read_to_end(&mut buffer).await.map_err(to_anyhow)?;
yield Ok(bytes::Bytes::from(buffer)) as anyhow::Result<bytes::Bytes>;
}
yield Ok(bytes::Bytes::from(logs))
};
return Some(Ok(Body::from_stream(stream)));
}
}
return None;
}
async fn get_job_logs(
Extension(db): Extension<DB>,
Path((w_id, id)): Path<(String, Uuid)>,
@@ -710,7 +750,11 @@ async fn get_job_logs(
if let Some(record) = record {
let logs = record.logs.unwrap_or_default();
#[cfg(all(feature = "enterprise", feature = "parquet"))]
if let Some(r) = get_logs_from_store(record.log_offset, &logs, record.log_file_index).await
if let Some(r) = get_logs_from_store(record.log_offset, &logs, &record.log_file_index).await
{
return r;
}
if let Some(r) = get_logs_from_disk(record.log_offset, &logs, &record.log_file_index).await
{
return r;
}
@@ -730,7 +774,10 @@ async fn get_job_logs(
let logs = text.logs.unwrap_or_default();
#[cfg(all(feature = "enterprise", feature = "parquet"))]
if let Some(r) = get_logs_from_store(text.log_offset, &logs, text.log_file_index).await {
if let Some(r) = get_logs_from_store(text.log_offset, &logs, &text.log_file_index).await {
return r;
}
if let Some(r) = get_logs_from_disk(text.log_offset, &logs, &text.log_file_index).await {
return r;
}
Ok(Body::from(logs))
@@ -2077,7 +2124,9 @@ enum PreviewKind {
Identity,
Http,
Noop,
Bundle,
}
#[derive(Deserialize)]
struct Preview {
content: Option<String>,
@@ -2420,6 +2469,7 @@ pub async fn run_workflow_as_code(
let (job_payload, tag, _delete_after_use, timeout) = match job.job_kind {
JobKind::Preview => (
JobPayload::Code(RawCode {
hash: None,
content: job.raw_code.unwrap_or_default(),
path: job.script_path,
language: job.language.unwrap_or_else(|| ScriptLang::Deno),
@@ -2521,7 +2571,7 @@ async fn run_wait_result(
w_id: String,
node_id_for_empty_return: Option<String>,
) -> error::Result<Response> {
let mut result;
let mut result = None;
let timeout = SERVER_CONFIG.read().await.timeout_wait_result.clone();
let timeout_ms = if timeout <= 0 {
2000
@@ -2545,7 +2595,9 @@ async fn run_wait_result(
)
.await
.ok();
} else {
}
if result.is_none() {
let row = sqlx::query(
"SELECT result, language, flow_status FROM completed_job WHERE id = $1 AND workspace_id = $2",
)
@@ -2563,8 +2615,6 @@ async fn run_wait_result(
FormattedResult::RawValue(rv) => rv,
FormattedResult::Vec(v) => Some(to_raw_value(&v)),
};
} else {
result = None;
}
}
@@ -3083,6 +3133,7 @@ async fn run_preview_script(
Some(PreviewKind::Identity) => JobPayload::Identity,
Some(PreviewKind::Noop) => JobPayload::Noop,
_ => JobPayload::Code(RawCode {
hash: None,
content: preview.content.unwrap_or_default(),
path: preview.path,
language: preview.language.unwrap_or(ScriptLang::Deno),
@@ -3117,6 +3168,136 @@ async fn run_preview_script(
Ok((StatusCode::CREATED, uuid.to_string()))
}
#[cfg(all(feature = "enterprise", feature = "parquet"))]
async fn run_bundle_preview_script(
authed: ApiAuthed,
Extension(db): Extension<DB>,
Extension(user_db): Extension<UserDB>,
Extension(rsmq): Extension<Option<rsmq_async::MultiplexedRsmq>>,
Path(w_id): Path<String>,
Query(run_query): Query<RunJobQuery>,
mut multipart: axum::extract::Multipart,
) -> error::Result<(StatusCode, String)> {
check_license_key_valid().await?;
check_scopes(&authed, || format!("runscript"))?;
if authed.is_operator {
return Err(error::Error::NotAuthorized(
"Operators cannot run preview jobs for security reasons".to_string(),
));
}
let mut job_id = None;
let mut tx = None;
let mut uploaded = false;
while let Some(field) = multipart.next_field().await.unwrap() {
let name = field.name().unwrap().to_string();
let data = field.bytes().await.unwrap();
if name == "preview" {
let preview: Preview = serde_json::from_slice(&data).map_err(to_anyhow)?;
let scheduled_for = run_query.get_scheduled_for(&db).await?;
let tag = run_query.tag.clone().or(preview.tag.clone());
check_tag_available_for_workspace(&w_id, &tag).await?;
let ltx =
PushIsolationLevel::Isolated(user_db.clone(), authed.clone().into(), rsmq.clone());
let args = preview.args.unwrap_or_default();
// hmap.insert("")
let (uuid, ntx) = push(
&db,
ltx,
&w_id,
match preview.kind {
Some(PreviewKind::Identity) => JobPayload::Identity,
Some(PreviewKind::Noop) => JobPayload::Noop,
_ => JobPayload::Code(RawCode {
hash: Some(PREVIEW_IS_CODEBASE_HASH),
content: preview.content.unwrap_or_default(),
path: preview.path,
language: preview.language.unwrap_or(ScriptLang::Deno),
lock: preview.lock,
concurrent_limit: None, // TODO(gbouv): once I find out how to store limits in the content of a script, should be easy to plug limits here
concurrency_time_window_s: None, // TODO(gbouv): same as above
cache_ttl: None,
dedicated_worker: preview.dedicated_worker,
}),
},
args,
authed.display_username(),
&authed.email,
username_to_permissioned_as(&authed.username),
scheduled_for,
None,
None,
None,
run_query.job_id,
false,
false,
None,
true,
tag,
run_query.timeout,
None,
None,
)
.await?;
job_id = Some(uuid);
tx = Some(ntx);
}
if name == "file" {
let id = job_id
.as_ref()
.ok_or_else(|| {
Error::BadRequest(
"script need to be passed first in the multipart upload".to_string(),
)
})?
.to_string();
uploaded = true;
if let Some(os) = windmill_common::s3_helpers::OBJECT_STORE_CACHE_SETTINGS
.read()
.await
.clone()
{
let path = windmill_common::s3_helpers::bundle(&w_id, &id);
if let Err(e) = os
.put(&object_store::path::Path::from(path.clone()), data)
.await
{
tracing::info!("Failed to put snapshot to s3 at {path}: {:?}", e);
return Err(Error::ExecutionErr(format!("Failed to put {path} to s3")));
}
} else {
return Err(Error::BadConfig("Object store is required for snapshot script and is not configured for servers".to_string()));
}
}
// println!("Length of `{}` is {} bytes", name, data.len());
}
if !uploaded {
return Err(Error::BadRequest("No file uploaded".to_string()));
}
if job_id.is_none() {
return Err(Error::BadRequest(
"No script found in the uploaded file".to_string(),
));
}
tx.unwrap().commit().await?;
Ok((StatusCode::CREATED, job_id.unwrap().to_string()))
}
#[cfg(not(all(feature = "enterprise", feature = "parquet")))]
async fn run_bundle_preview_script() -> error::Result<(StatusCode, String)> {
return Err(Error::BadRequest(
"bundle preview is an ee feature".to_string(),
));
}
#[derive(Deserialize)]
pub struct RunDependenciesRequest {
pub raw_scripts: Vec<RawScriptForDependencies>,
@@ -3511,45 +3692,20 @@ pub struct JobUpdate {
pub flow_status: Option<serde_json::Value>,
}
// #[cfg(all(feature = "enterprise", feature = "parquet"))]
// async fn get_logs_from_store(
// log_offset: i32,
// logs: &str,
// log_file_index: Option<Vec<String>>,
// ) -> Option<error::Result<Body>> {
// if log_offset > 0 {
// if let Some(file_index) = log_file_index {
// tracing::debug!("Getting logs from store: {file_index:?}");
// if let Some(os) = OBJECT_STORE_CACHE_SETTINGS.read().await.clone() {
// tracing::debug!("object store client present, streaming from there");
// let logs = logs.to_string();
// let stream = async_stream::stream! {
// for file_p in file_index {
// let file_p_2 = file_p.clone();
// let file = os.get(&object_store::path::Path::from(file_p)).await;
// if let Ok(file) = file {
// if let Ok(bytes) = file.bytes().await {
// yield Ok(bytes::Bytes::from(bytes)) as object_store::Result<bytes::Bytes>;
// }
// } else {
// tracing::debug!("error getting file from store: {file_p_2}: {}", file.err().unwrap());
// }
// }
// yield Ok(bytes::Bytes::from(logs))
// };
// return Some(Ok(Body::from_stream(stream)));
// } else {
// tracing::debug!("object store client not present, cannot stream logs from store");
// }
// }
// }
// return None;
// }
#[cfg(all(feature = "enterprise", feature = "parquet"))]
async fn get_log_file(Path((_w_id, file_p)): Path<(String, String)>) -> error::Result<Response> {
let local_file = format!("{TMP_DIR}/logs/{file_p}");
if tokio::fs::metadata(&local_file).await.is_ok() {
let mut file = tokio::fs::File::open(local_file).await.map_err(to_anyhow)?;
let mut buffer = Vec::new();
file.read_to_end(&mut buffer).await.map_err(to_anyhow)?;
let res = Response::builder()
.header(http::header::CONTENT_TYPE, "text/plain")
.body(Body::from(bytes::Bytes::from(buffer)))
.unwrap();
return Ok(res);
}
#[cfg(all(feature = "enterprise", feature = "parquet"))]
if let Some(os) = OBJECT_STORE_CACHE_SETTINGS.read().await.clone() {
let file = os
.get(&object_store::path::Path::from(format!("logs/{file_p}")))
@@ -3579,12 +3735,9 @@ async fn get_log_file(Path((_w_id, file_p)): Path<(String, String)>) -> error::R
"Object store client not present, cannot stream logs from store".to_string(),
));
}
}
#[cfg(not(all(feature = "enterprise", feature = "parquet")))]
async fn get_log_file(Path((_w_id, file_p)): Path<(String, String)>) -> error::Result<Response> {
return Err(error::Error::NotFound(format!(
"Get log file is an EE feature: {}",
"File not found on server logs volume /tmp/windmill/logs and no distributed logs s3 storage for {}",
file_p
)));
}
backend/windmill-api/src/lib.rs
+21 -7
View File
@@ -25,6 +25,7 @@ use db::DB;
use git_version::git_version;
use reqwest::Client;
use std::collections::HashMap;
use std::time::Duration;
use std::{net::SocketAddr, sync::Arc};
use tokio::sync::RwLock;
use tower::ServiceBuilder;
@@ -104,6 +105,8 @@ lazy_static::lazy_static! {
pub static ref HTTP_CLIENT: Client = reqwest::ClientBuilder::new()
.user_agent("windmill/beta")
.connect_timeout(Duration::from_secs(10))
.timeout(Duration::from_secs(30))
.danger_accept_invalid_certs(std::env::var("ACCEPT_INVALID_CERTS").is_ok())
.build().unwrap();
@@ -141,14 +144,12 @@ pub async fn run_server(
));
let argon2 = Arc::new(Argon2::default());
let disable_response_logs = std::env::var("DISABLE_RESPONSE_LOGS")
.ok()
.map(|x| x == "true")
.unwrap_or(false);
let middleware_stack = ServiceBuilder::new()
.layer(
TraceLayer::new_for_http()
.on_response(MyOnResponse {})
.make_span_with(MyMakeSpan {})
.on_request(())
.on_failure(MyOnFailure {}),
)
.layer(Extension(db.clone()))
.layer(Extension(rsmq))
.layer(Extension(user_db))
@@ -285,6 +286,18 @@ pub async fn run_server(
.fallback(static_assets::static_handler)
.layer(middleware_stack);
let app = if disable_response_logs {
app
} else {
app.layer(
TraceLayer::new_for_http()
.on_response(MyOnResponse {})
.make_span_with(MyMakeSpan {})
.on_request(())
.on_failure(MyOnFailure {}),
)
};
let instance_name = rd_string(5);
let listener = tokio::net::TcpListener::bind(addr).await.unwrap();
@@ -320,6 +333,7 @@ async fn is_up_to_date() -> Result<String, AppError> {
let error_reading_version = || anyhow::anyhow!("Error reading latest released version");
let version = HTTP_CLIENT
.get("https://api.github.com/repos/windmill-labs/windmill/releases/latest")
.timeout(Duration::from_secs(10))
.send()
.await
.context("Impossible to reach api.github")?
backend/windmill-api/src/scripts.rs
+109 -19
View File
@@ -13,6 +13,9 @@ use crate::{
webhook_util::{WebhookMessage, WebhookShared},
HTTP_CLIENT,
};
#[cfg(all(feature = "enterprise", feature = "parquet"))]
use axum::extract::Multipart;
use axum::{
extract::{Extension, Path, Query},
response::IntoResponse,
@@ -31,6 +34,10 @@ use std::{
};
use windmill_audit::audit_ee::audit_log;
use windmill_audit::ActionKind;
#[cfg(all(feature = "enterprise", feature = "parquet"))]
use windmill_common::error::to_anyhow;
use windmill_common::{
db::UserDB,
error::{Error, JsonResult, Result},
@@ -116,6 +123,7 @@ pub fn workspaced_service() -> Router {
.route("/list", get(list_scripts))
.route("/list_search", get(list_search_scripts))
.route("/create", post(create_script))
.route("/create_snapshot", post(create_snapshot_script))
.route("/archive/p/*path", post(archive_script_by_path))
.route("/get/draft/*path", get(get_script_by_path_w_draft))
.route("/get/p/*path", get(get_script_by_path))
@@ -196,6 +204,7 @@ async fn list_scripts(
"draft_only",
"ws_error_handler_muted",
"no_main_func",
"codebase IS NOT NULL as use_codebase"
])
.left()
.join("favorite")
@@ -312,6 +321,82 @@ fn hash_script(ns: &NewScript) -> i64 {
dh.finish() as i64
}
#[cfg(not(all(feature = "enterprise", feature = "parquet")))]
async fn create_snapshot_script() -> Result<(StatusCode, String)> {
Err(Error::BadRequest("Upgrade to EE to use bundle".to_string()))
}
#[cfg(all(feature = "enterprise", feature = "parquet"))]
async fn create_snapshot_script(
authed: ApiAuthed,
Extension(user_db): Extension<UserDB>,
Extension(rsmq): Extension<Option<rsmq_async::MultiplexedRsmq>>,
Extension(webhook): Extension<WebhookShared>,
Extension(db): Extension<DB>,
Path(w_id): Path<String>,
mut multipart: Multipart,
) -> Result<(StatusCode, String)> {
let mut script_hash = None;
let mut tx = None;
let mut uploaded = false;
while let Some(field) = multipart.next_field().await.unwrap() {
let name = field.name().unwrap().to_string();
let data = field.bytes().await.unwrap();
if name == "script" {
let ns = Some(serde_json::from_slice(&data).map_err(to_anyhow)?);
let (new_hash, ntx) = create_script_internal(
ns.unwrap(),
w_id.clone(),
authed.clone(),
db.clone(),
rsmq.clone(),
user_db.clone(),
webhook.clone(),
)
.await?;
script_hash = Some(new_hash.to_string());
tx = Some(ntx);
}
if name == "file" {
let hash = script_hash.as_ref().ok_or_else(|| {
Error::BadRequest(
"script need to be passed first in the multipart upload".to_string(),
)
})?;
uploaded = true;
if let Some(os) = windmill_common::s3_helpers::OBJECT_STORE_CACHE_SETTINGS
.read()
.await
.clone()
{
let path = windmill_common::s3_helpers::bundle(&w_id, &hash);
if let Err(e) = os
.put(&object_store::path::Path::from(path.clone()), data)
.await
{
tracing::info!("Failed to put snapshot to s3 at {path}: {:?}", e);
return Err(Error::ExecutionErr(format!("Failed to put {path} to s3")));
}
} else {
return Err(Error::BadConfig("Object store is required for snapshot script and is not configured for servers".to_string()));
}
}
// println!("Length of `{}` is {} bytes", name, data.len());
}
if !uploaded {
return Err(Error::BadRequest("No file uploaded".to_string()));
}
if script_hash.is_none() {
return Err(Error::BadRequest(
"No script found in the uploaded file".to_string(),
));
}
tx.unwrap().commit().await?;
return Ok((StatusCode::CREATED, format!("{}", script_hash.unwrap())));
}
async fn create_script(
authed: ApiAuthed,
Extension(user_db): Extension<UserDB>,
@@ -321,6 +406,24 @@ async fn create_script(
Path(w_id): Path<String>,
Json(ns): Json<NewScript>,
) -> Result<(StatusCode, String)> {
let (hash, tx) = create_script_internal(ns, w_id, authed, db, rsmq, user_db, webhook).await?;
tx.commit().await?;
Ok((StatusCode::CREATED, format!("{}", hash)))
}
async fn create_script_internal<'c>(
ns: NewScript,
w_id: String,
authed: ApiAuthed,
db: sqlx::Pool<Postgres>,
rsmq: Option<rsmq_async::MultiplexedRsmq>,
user_db: UserDB,
webhook: WebhookShared,
) -> Result<(
ScriptHash,
QueueTransaction<'c, rsmq_async::MultiplexedRsmq>,
)> {
let codebase = ns.codebase.as_ref();
#[cfg(not(feature = "enterprise"))]
if ns.ws_error_handler_muted.is_some_and(|val| val) {
return Err(Error::BadRequest(
@@ -328,12 +431,10 @@ async fn create_script(
.to_string(),
));
}
let script_path = ns.path.clone();
let hash = ScriptHash(hash_script(&ns));
let authed = maybe_refresh_folders(&ns.path, &w_id, authed, &db).await;
let mut tx: QueueTransaction<'_, _> = (rsmq.clone(), user_db.begin(&authed).await?).into();
if sqlx::query_scalar!(
"SELECT 1 FROM script WHERE hash = $1 AND workspace_id = $2",
hash.0,
@@ -349,7 +450,6 @@ async fn create_script(
.to_owned(),
));
};
let clashing_script = sqlx::query_as::<_, Script>(
"SELECT * FROM script WHERE path = $1 AND archived = false AND workspace_id = $2",
)
@@ -357,7 +457,6 @@ async fn create_script(
.bind(&w_id)
.fetch_optional(&mut tx)
.await?;
struct ParentInfo {
p_hashes: Vec<i64>,
perms: serde_json::Value,
@@ -450,13 +549,11 @@ async fn create_script(
r
}
}?;
let p_hashes = parent_hashes_and_perms.as_ref().map(|v| &v.p_hashes[..]);
let extra_perms = parent_hashes_and_perms
.as_ref()
.map(|v| v.perms.clone())
.unwrap_or(json!({}));
let lock = if !(ns.language == ScriptLang::Python3
|| ns.language == ScriptLang::Go
|| ns.language == ScriptLang::Bun
@@ -467,23 +564,20 @@ async fn create_script(
ns.lock
.and_then(|e| if e.is_empty() { None } else { Some(e) })
};
let needs_lock_gen = lock.is_none();
let envs = ns.envs.as_ref().map(|x| x.as_slice());
let envs = if ns.envs.is_none() || ns.envs.as_ref().unwrap().is_empty() {
None
} else {
envs
};
//::text::json is to ensure we use serde_json with preserve order
sqlx::query!(
"INSERT INTO script (workspace_id, hash, path, parent_hashes, summary, description, \
content, created_by, schema, is_template, extra_perms, lock, language, kind, tag, \
draft_only, envs, concurrent_limit, concurrency_time_window_s, cache_ttl, \
dedicated_worker, ws_error_handler_muted, priority, restart_unless_cancelled, \
delete_after_use, timeout, concurrency_key, visible_to_runner_only, no_main_func) \
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9::text::json, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22, $23, $24, $25, $26, $27, $28, $29)",
delete_after_use, timeout, concurrency_key, visible_to_runner_only, no_main_func, codebase) \
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9::text::json, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22, $23, $24, $25, $26, $27, $28, $29, $30)",
&w_id,
&hash.0,
ns.path,
@@ -512,11 +606,11 @@ async fn create_script(
ns.timeout,
ns.concurrency_key,
ns.visible_to_runner_only,
ns.no_main_func
ns.no_main_func,
codebase
)
.execute(&mut tx)
.await?;
let p_path_opt = parent_hashes_and_perms.as_ref().map(|x| x.p_path.clone());
if let Some(ref p_path) = p_path_opt {
sqlx::query!(
@@ -566,7 +660,6 @@ async fn create_script(
.execute(&mut tx)
.await?;
}
if p_hashes.is_some() && !p_hashes.unwrap().is_empty() {
audit_log(
&mut tx,
@@ -612,7 +705,6 @@ async fn create_script(
},
);
}
let permissioned_as = username_to_permissioned_as(&authed.username);
if needs_lock_gen {
let tag = if ns.dedicated_worker.is_some_and(|x| x) {
@@ -661,7 +753,7 @@ async fn create_script(
None,
)
.await?;
new_tx.commit().await?;
Ok((hash, new_tx))
} else {
handle_deployment_metadata(
&authed.email,
@@ -678,10 +770,8 @@ async fn create_script(
false,
)
.await?;
tx.commit().await?;
Ok((hash, tx))
}
Ok((StatusCode::CREATED, format!("{}", hash)))
}
pub async fn get_hub_script_by_path(
backend/windmill-api/src/webhook_util.rs
+1
View File
@@ -71,6 +71,7 @@ impl WebhookShared {
let (tx, mut rx) = mpsc::unbounded_channel::<WebhookPayload>();
let _process = tokio::spawn(async move {
let client = reqwest::Client::builder()
.connect_timeout(Duration::from_secs(5))
// TODO: investigate pool timeouts and such if TCP load is high
.timeout(Duration::from_secs(5))
.build()
backend/windmill-api/src/workers.rs
+47 -5
View File
@@ -14,14 +14,16 @@ use axum::{
use serde::{Deserialize, Serialize};
use sqlx::FromRow;
use uuid::Uuid;
use windmill_common::{
db::UserDB,
error::JsonResult,
utils::{paginate, Pagination},
worker::{ALL_TAGS, DEFAULT_TAGS, DEFAULT_TAGS_PER_WORKSPACE},
DB,
};
use crate::db::ApiAuthed;
use crate::{db::ApiAuthed, utils::require_super_admin};
pub fn global_service() -> Router {
Router::new()
@@ -33,6 +35,7 @@ pub fn global_service() -> Router {
get(get_default_tags_per_workspace),
)
.route("/get_default_tags", get(get_default_tags))
.route("/queue_metrics", get(get_queue_metrics))
}
#[derive(FromRow, Serialize, Deserialize)]
@@ -43,9 +46,12 @@ struct WorkerPing {
started_at: chrono::DateTime<chrono::Utc>,
ip: String,
jobs_executed: i32,
current_job_id: Option<Uuid>,
current_job_workspace_id: Option<String>,
custom_tags: Option<Vec<String>>,
worker_group: String,
wm_version: String,
occupancy_rate: Option<f32>,
}
#[derive(Serialize, Deserialize)]
@@ -62,21 +68,25 @@ pub struct ListWorkerQuery {
async fn list_worker_pings(
authed: ApiAuthed,
Extension(db): Extension<DB>,
Extension(user_db): Extension<UserDB>,
Query(query): Query<ListWorkerQuery>,
) -> JsonResult<Vec<WorkerPing>> {
let is_super_admin = require_super_admin(&db, &authed.email).await.is_ok();
let mut tx = user_db.begin(&authed).await?;
let (per_page, offset) = paginate(Pagination { page: query.page, per_page: query.per_page });
let rows = sqlx::query_as!(
WorkerPing,
"SELECT worker, worker_instance, EXTRACT(EPOCH FROM (now() - ping_at))::integer as last_ping, started_at, ip, jobs_executed, custom_tags, worker_group, wm_version FROM worker_ping
WHERE ($1::integer IS NULL AND ping_at > now() - interval '5 minute') OR (ping_at > now() - ($1 || ' seconds')::interval)
ORDER BY ping_at desc LIMIT $2 OFFSET $3",
"SELECT worker, worker_instance, EXTRACT(EPOCH FROM (now() - ping_at))::integer as last_ping, started_at, ip, jobs_executed, CASE WHEN $4 IS TRUE THEN current_job_id ELSE NULL END as current_job_id, CASE WHEN $4 IS TRUE THEN current_job_workspace_id ELSE NULL END as current_job_workspace_id, custom_tags, worker_group, wm_version, occupancy_rate
FROM worker_ping
WHERE ($1::integer IS NULL AND ping_at > now() - interval '5 minute') OR (ping_at > now() - ($1 || ' seconds')::interval)
ORDER BY ping_at desc LIMIT $2 OFFSET $3",
query.ping_since,
per_page as i64,
offset as i64
offset as i64,
is_super_admin
)
.fetch_all(&mut *tx)
.await?;
@@ -118,3 +128,35 @@ async fn get_default_tags_per_workspace() -> JsonResult<bool> {
async fn get_default_tags() -> JsonResult<Vec<String>> {
Ok(Json(DEFAULT_TAGS.clone()))
}
#[derive(Serialize)]
struct QueueMetric {
id: String,
values: Vec<serde_json::Value>,
}
async fn get_queue_metrics(
authed: ApiAuthed,
Extension(db): Extension<DB>,
) -> JsonResult<Vec<QueueMetric>> {
require_super_admin(&db, &authed.email).await?;
let queue_metrics = sqlx::query_as!(
QueueMetric,
"WITH queue_metrics as (
SELECT id, value, created_at
FROM metrics
WHERE id LIKE 'queue_%'
AND created_at > now() - interval '14 day'
ORDER BY created_at ASC
)
SELECT id, array_agg(json_build_object('value', value, 'created_at', created_at)) as \"values!\"
FROM queue_metrics
GROUP BY id
ORDER BY id ASC"
)
.fetch_all(&db)
.await?;
Ok(Json(queue_metrics))
}
backend/windmill-api/src/workspaces.rs
+3
View File
@@ -2218,6 +2218,8 @@ struct ScriptMetadata {
pub visible_to_runner_only: Option<bool>,
#[serde(skip_serializing_if = "Option::is_none")]
pub no_main_func: Option<bool>,
#[serde(skip_serializing_if = "Option::is_none")]
pub codebase: Option<String>,
}
pub fn is_none_or_false(val: &Option<bool>) -> bool {
@@ -2495,6 +2497,7 @@ async fn tarball_workspace(
restart_unless_cancelled: script.restart_unless_cancelled,
visible_to_runner_only: script.visible_to_runner_only,
no_main_func: script.no_main_func,
codebase: script.codebase,
};
let metadata_str = serde_json::to_string_pretty(&metadata).unwrap();
archive
backend/windmill-common/Cargo.toml
+3 -2
View File
@@ -10,7 +10,7 @@ enterprise = []
prometheus = ["dep:prometheus"]
flamegraph = ["dep:tracing-flame"]
loki = ["dep:tracing-loki"]
parquet = ["dep:object_store", "dep:aws-config", "dep:aws-sdk-sts"]
parquet = ["dep:object_store", "dep:aws-config", "dep:aws-sdk-sts", "dep:bytes"]
[lib]
name = "windmill_common"
@@ -47,4 +47,5 @@ prometheus = { workspace = true, optional = true }
aws-config = { workspace = true, optional = true }
aws-sdk-sts = { workspace = true, optional = true }
indexmap.workspace = true
mail-send.workspace = true
bytes = { workspace = true, optional = true }
mail-send.workspace = true
backend/windmill-common/src/error.rs
+2
View File
@@ -58,6 +58,8 @@ pub enum Error {
JsonErr(serde_json::Value),
#[error("{0}")]
OpenAIError(String),
#[error("{0}")]
AlreadyCompleted(String),
}
impl Error {
backend/windmill-common/src/external_ip.rs
+13 -10
View File
@@ -13,16 +13,19 @@
use std::time::Duration;
use reqwest::Result;
pub async fn get_ip() -> Result<String> {
reqwest::ClientBuilder::new()
.timeout(Duration::from_secs(3))
pub async fn get_ip() -> anyhow::Result<String> {
tokio::select! {
biased;
_ = tokio::time::sleep(Duration::from_secs(10)) => {
return Err(anyhow::anyhow!("Expected to get ip under 10s"))
},
ip = reqwest::ClientBuilder::new()
.connect_timeout(Duration::from_secs(5))
.timeout(Duration::from_secs(5))
.build()?
.get("https://hub.windmill.dev/getip")
.send()
.await?
.error_for_status()?
.text()
.await
.send() => Ok(ip?
.error_for_status()?
.text().await?),
}
}
backend/windmill-common/src/jobs.rs
+1
View File
@@ -345,6 +345,7 @@ pub enum JobPayload {
pub struct RawCode {
pub content: String,
pub path: Option<String>,
pub hash: Option<i64>,
pub language: ScriptLang,
pub lock: Option<String>,
pub concurrent_limit: Option<i32>,
backend/windmill-common/src/s3_helpers.rs
+46
View File
@@ -177,6 +177,48 @@ pub fn build_object_store_client(
}
}
#[cfg(feature = "parquet")]
pub async fn attempt_fetch_bytes(
client: Arc<dyn ObjectStore>,
path: &str,
) -> error::Result<bytes::Bytes> {
use object_store::path::Path;
let object = client.get(&Path::from(path)).await;
if let Err(e) = object {
tracing::info!(
"Failed to pull bytes from object store at path {path}. Error: {:?}",
e
);
return Err(error::Error::ExecutionErr(format!(
"Failed to pull bytes from object store: {path}"
)));
}
let bytes = object.unwrap().bytes().await;
if bytes.is_err() {
tracing::info!(
"Failed to read bytes from object store: {path}. Error: {:?}",
bytes.err()
);
return Err(error::Error::ExecutionErr(format!(
"Failed to read bytes from object store: {path}"
)));
}
let bytes = bytes.unwrap();
tracing::info!("{path} len: {}", bytes.len());
if bytes.len() == 0 {
tracing::info!("object {path} not found in bucket, bytes empty",);
return Err(error::Error::ExecutionErr(format!(
"object {path} does not exist in bucket"
)));
}
return Ok(bytes);
}
#[cfg(feature = "parquet")]
use aws_config::{default_provider::credentials::DefaultCredentialsChain, Region};
#[cfg(feature = "parquet")]
@@ -408,3 +450,7 @@ impl CredentialProvider for AwsCredentialAdapter {
}))
}
}
pub fn bundle(w_id: &str, hash: &str) -> String {
format!("script_bundle/{}/{}", w_id, hash)
}
backend/windmill-common/src/scripts.rs
+11
View File
@@ -128,6 +128,8 @@ impl Display for ScriptKind {
}
}
pub const PREVIEW_IS_CODEBASE_HASH: i64 = -42;
#[derive(Serialize, sqlx::FromRow)]
pub struct Script {
pub workspace_id: String,
@@ -177,6 +179,8 @@ pub struct Script {
pub visible_to_runner_only: Option<bool>,
#[serde(skip_serializing_if = "Option::is_none")]
pub no_main_func: Option<bool>,
#[serde(skip_serializing_if = "Option::is_none")]
pub codebase: Option<String>,
}
#[derive(Serialize, sqlx::FromRow)]
@@ -198,6 +202,12 @@ pub struct ListableScript {
pub ws_error_handler_muted: Option<bool>,
#[serde(skip_serializing_if = "Option::is_none")]
pub no_main_func: Option<bool>,
#[serde(skip_serializing_if = "is_false")]
pub use_codebase: bool,
}
fn is_false(x: &bool) -> bool {
return !x;
}
#[derive(Serialize)]
@@ -256,6 +266,7 @@ pub struct NewScript {
pub concurrency_key: Option<String>,
pub visible_to_runner_only: Option<bool>,
pub no_main_func: Option<bool>,
pub codebase: Option<String>,
}
fn lock_deserialize<'de, D>(deserializer: D) -> Result<Option<String>, D::Error>
backend/windmill-common/src/utils.rs
+4 -1
View File
@@ -6,7 +6,9 @@
* LICENSE-AGPL for a copy of the license.
*/
use crate::ee::{trigger_critical_error_channels, LICENSE_KEY_ID};
#[cfg(feature = "enterprise")]
use crate::ee::trigger_critical_error_channels;
use crate::ee::LICENSE_KEY_ID;
use crate::error::{to_anyhow, Error, Result};
use crate::global_settings::UNIQUE_ID_SETTING;
use crate::server::Smtp;
@@ -233,5 +235,6 @@ pub async fn send_email(
pub async fn report_critical_error(error_message: String) -> () {
tracing::error!("CRITICAL ERROR: {error_message}");
#[cfg(feature = "enterprise")]
trigger_critical_error_channels(error_message).await;
}
backend/windmill-common/src/variables.rs
-6
View File
@@ -276,12 +276,6 @@ pub async fn get_reserved_variables(
description: "State resource path unique to a script and its trigger".to_string(),
is_custom: false,
},
ContextualVariable {
name: "WM_STATE_PATH_NEW".to_string(),
value: state_path,
description: "State resource path unique to a script and its trigger (legacy)".to_string(),
is_custom: false,
},
ContextualVariable {
name: "WM_FLOW_STEP_ID".to_string(),
value: step_id.unwrap_or_else(|| "".to_string()),
backend/windmill-common/src/worker.rs
+43 -2
View File
@@ -70,6 +70,8 @@ lazy_static::lazy_static! {
}
pub const TMP_DIR: &str = "/tmp/windmill";
pub async fn reload_custom_tags_setting(db: &DB) -> error::Result<()> {
let q = sqlx::query!(
"SELECT value FROM global_settings WHERE name = $1",
@@ -140,15 +142,54 @@ pub async fn update_ping(worker_instance: &str, worker_name: &str, ip: &str, db:
.map(|x| format!("{}:{}", x.workspace_id, x.path)),
)
};
let mut vcpus = std::process::Command::new("cat")
.args(["/sys/fs/cgroup/cpu.max"])
.output()
.ok()
.map(|o| {
String::from_utf8_lossy(&o.stdout)
.to_string()
.split(" ")
.map(|s| s.to_string())
.collect::<Vec<String>>()
.get(0)
.map(|s| s.to_string())
})
.flatten();
if vcpus.is_none() {
vcpus = std::process::Command::new("cat")
.args(["/sys/fs/cgroup/cpu/cpu.cfs_quota_us"])
.output()
.ok()
.map(|o| String::from_utf8_lossy(&o.stdout).to_string())
}
let mut memory = std::process::Command::new("cat")
.args(["/sys/fs/cgroup/memory.max"])
.output()
.ok()
.map(|o| String::from_utf8_lossy(&o.stdout).to_string());
if memory.is_none() {
memory = std::process::Command::new("cat")
.args(["/sys/fs/cgroup/memory/memory.limit_in_bytes"])
.output()
.ok()
.map(|o| String::from_utf8_lossy(&o.stdout).to_string())
}
sqlx::query!(
"INSERT INTO worker_ping (worker_instance, worker, ip, custom_tags, worker_group, dedicated_worker, wm_version) VALUES ($1, $2, $3, $4, $5, $6, $7) ON CONFLICT (worker) DO UPDATE set ip = $3, custom_tags = $4, worker_group = $5",
"INSERT INTO worker_ping (worker_instance, worker, ip, custom_tags, worker_group, dedicated_worker, wm_version, vcpus, memory) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9) ON CONFLICT (worker) DO UPDATE set ip = $3, custom_tags = $4, worker_group = $5",
worker_instance,
worker_name,
ip,
tags.as_slice(),
*WORKER_GROUP,
dw,
crate::utils::GIT_VERSION
crate::utils::GIT_VERSION,
vcpus.map(|x| x.parse::<i64>().ok()).flatten(),
memory.map(|x| x.parse::<i64>().ok()).flatten()
)
.execute(db)
.await
backend/windmill-queue/src/jobs.rs
+97 -33
View File
@@ -106,11 +106,10 @@ lazy_static::lazy_static! {
lazy_static::lazy_static! {
pub static ref HTTP_CLIENT: Client = reqwest::ClientBuilder::new()
.user_agent("windmill/beta")
.timeout(std::time::Duration::from_secs(20))
.connect_timeout(std::time::Duration::from_secs(10))
.build().unwrap();
pub static ref HTTP_CLIENT_WORKER: Client = reqwest::ClientBuilder::new()
.user_agent("windmill/beta")
.build().unwrap();
}
@@ -133,24 +132,16 @@ pub struct CanceledBy {
pub reason: Option<String>,
}
#[async_recursion]
pub async fn cancel_job<'c: 'async_recursion>(
pub async fn cancel_single_job<'c>(
username: &str,
reason: Option<String>,
id: Uuid,
job_running: &QueuedJob,
w_id: &str,
mut tx: Transaction<'c, Postgres>,
db: &Pool<Postgres>,
rsmq: Option<rsmq_async::MultiplexedRsmq>,
force_cancel: bool,
) -> error::Result<(Transaction<'c, Postgres>, Option<Uuid>)> {
let job_running = get_queued_job_tx(id, &w_id, &mut tx).await?;
if job_running.is_none() {
return Ok((tx, None));
}
let job_running = job_running.unwrap();
if ((job_running.running || job_running.root_job.is_some()) || (job_running.is_flow()))
&& !force_cancel
{
@@ -158,7 +149,7 @@ pub async fn cancel_job<'c: 'async_recursion>(
"UPDATE queue SET canceled = true, canceled_by = $1, canceled_reason = $2, scheduled_for = now(), suspend = 0 WHERE id = $3 AND workspace_id = $4 RETURNING id",
username,
reason,
id,
job_running.id,
w_id
)
.fetch_optional(&mut *tx)
@@ -172,7 +163,7 @@ pub async fn cancel_job<'c: 'async_recursion>(
.unwrap_or_else(|| "unexplicited reasons".to_string());
let e = serde_json::json!({"message": format!("Job canceled: {reason} by {username}"), "name": "Canceled", "reason": reason, "canceler": username});
append_logs(
id,
job_running.id,
w_id.to_string(),
format!("canceled by {username}: (force cancel: {force_cancel})"),
db,
@@ -180,7 +171,7 @@ pub async fn cancel_job<'c: 'async_recursion>(
.await;
let add_job = add_completed_job_error(
&db,
&job_running,
job_running,
job_running.mem_peak.unwrap_or(0),
Some(CanceledBy { username: Some(username.to_string()), reason: Some(reason) }),
e,
@@ -189,16 +180,73 @@ pub async fn cancel_job<'c: 'async_recursion>(
false,
)
.await;
if let Err(e) = add_job {
tracing::error!("Failed to add canceled job: {}", e);
}
}
if let Some(mut rsmq) = rsmq.clone() {
rsmq.change_message_visibility(&job_running.tag, &id.to_string(), 0)
rsmq.change_message_visibility(&job_running.tag, &job_running.id.to_string(), 0)
.await
.map_err(|e| anyhow::anyhow!(e))?;
}
Ok((tx, Some(job_running.id)))
}
pub async fn cancel_job<'c>(
username: &str,
reason: Option<String>,
id: Uuid,
w_id: &str,
mut tx: Transaction<'c, Postgres>,
db: &Pool<Postgres>,
rsmq: Option<rsmq_async::MultiplexedRsmq>,
force_cancel: bool,
) -> error::Result<(Transaction<'c, Postgres>, Option<Uuid>)> {
let job = get_queued_job_tx(id, &w_id, &mut tx).await?;
if job.is_none() {
return Ok((tx, None));
}
let job = job.unwrap();
let (ntx, _) = cancel_single_job(
username,
reason.clone(),
&job,
w_id,
tx,
db,
rsmq.clone(),
force_cancel,
)
.await?;
tx = ntx;
// soft cancel parent if force cancel and job is a flow step
if force_cancel && job.is_flow_step {
if let Some(parent_job_id) = job.parent_job {
let job = get_queued_job_tx(parent_job_id, &w_id, &mut tx).await?;
if let Some(job) = job {
let (ntx, _) = cancel_single_job(
username,
reason.clone(),
&job,
w_id,
tx,
db,
rsmq.clone(),
false,
)
.await?;
tx = ntx;
}
}
}
// cancel children
let mut jobs = vec![id];
let mut jobs_to_cancel = vec![];
while !jobs.is_empty() {
@@ -213,19 +261,23 @@ pub async fn cancel_job<'c: 'async_recursion>(
jobs.extend(new_jobs.clone());
jobs_to_cancel.extend(new_jobs);
}
for job in jobs_to_cancel {
let (ntx, _) = cancel_job(
username,
reason.clone(),
job,
w_id,
tx,
db,
rsmq.clone(),
force_cancel,
)
.await?;
tx = ntx;
for job_id in jobs_to_cancel {
let job = get_queued_job_tx(job_id, &w_id, &mut tx).await?;
if let Some(job) = job {
let (ntx, _) = cancel_single_job(
username,
reason.clone(),
&job,
w_id,
tx,
db,
rsmq.clone(),
force_cancel,
)
.await?;
tx = ntx;
}
}
Ok((tx, Some(id)))
}
@@ -633,8 +685,19 @@ pub async fn add_completed_job<
if let Some(schedule) = schedule {
skip_downstream_error_handlers = schedule.ws_error_handler_muted;
if !queued_job.is_flow() {
// script only
// script or flow that failed on start and might not have been rescheduled
let schedule_next_tick = !queued_job.is_flow()
|| {
let flow_status = queued_job.parse_flow_status();
flow_status.is_some_and(|fs| {
fs.step == 0
&& fs.modules.get(0).is_some_and(|m| {
matches!(m, FlowStatusModule::WaitingForPriorSteps { .. }) || matches!(m, FlowStatusModule::Failure { job, ..} if job == &Uuid::nil())
})
})
};
if schedule_next_tick {
if let Err(err) = handle_maybe_scheduled_job(
rsmq.clone(),
db,
@@ -2882,6 +2945,7 @@ pub async fn push<'c, T: Serialize + Send + Sync, R: rsmq_async::RsmqConnection
JobPayload::Code(RawCode {
content,
path,
hash,
language,
lock,
concurrent_limit,
@@ -2889,7 +2953,7 @@ pub async fn push<'c, T: Serialize + Send + Sync, R: rsmq_async::RsmqConnection
cache_ttl,
dedicated_worker,
}) => (
None,
hash,
path,
Some((content, lock)),
JobKind::Preview,
backend/windmill-worker/loader.bun.ts → backend/windmill-worker/loader.bun.js
+1 -1
View File
@@ -27,7 +27,7 @@ const p = {
const cdir = resolve("./");
const cdirNoPrivate = cdir.replace(/^\/private/, ""); // for macos
const filter = new RegExp(
`^(?!\\.\/main\\.ts)(?!${cdir}\/main\\.ts)(?!(?:/private)?${cdirNoPrivate}\/wrapper\\.ts).*\\.ts$`
`^(?!\\.\/main\\.ts)(?!${cdir}\/main\\.ts)(?!(?:/private)?${cdirNoPrivate}\/wrapper\\.mjs).*\\.ts$`
);
build.onResolve({ filter }, (args) => {
const file_path =
backend/windmill-worker/loader_builder.bun.ts → backend/windmill-worker/loader_builder.bun.js
+3 -3
View File
@@ -21,7 +21,7 @@ if (!bo.success) {
content.replaceAll("__require", "require")
);
const dependencies: Record<string, string[]> = {};
const dependencies = {};
for (const i of imports) {
let [_, name, version] = i.path.match(captureVersion) ?? [];
if (name == undefined) {
@@ -46,7 +46,7 @@ if (!bo.success) {
}
}
}
const resolvedDeps: Record<string, string> = {};
const resolvedDeps = {};
for (const i in dependencies) {
const versions = dependencies[i];
resolvedDeps[i] =
@@ -61,7 +61,7 @@ if (!bo.success) {
JSON.stringify({ dependencies: resolvedDeps }, null, 2)
);
function reduceIntersect(versions: string[], name: string): string {
function reduceIntersect(versions, name) {
console.log(
`multiple versions detected for ${name}: ${versions.join(", ")}`
);
backend/windmill-worker/nsjail/run.bun.config.proto
+8 -8
View File
@@ -66,7 +66,7 @@ mount {
src: "{JOB_DIR}/package.json"
dst: "/tmp/{LANG}/package.json"
is_bind: true
mandatory: true
mandatory: false
}
mount {
@@ -77,13 +77,6 @@ mount {
}
mount {
src: "{JOB_DIR}/wrapper.ts"
dst: "/tmp/{LANG}/wrapper.ts"
is_bind: true
mandatory: false
}
mount {
src: "{JOB_DIR}/wrapper.mjs"
dst: "/tmp/{LANG}/wrapper.mjs"
@@ -114,6 +107,13 @@ mount {
mandatory: false
}
mount {
src: "{JOB_DIR}/main.js"
dst: "/tmp/{LANG}/main.js"
is_bind: true
mandatory: false
}
mount {
src: "{JOB_DIR}/args.json"
backend/windmill-worker/src/bun_executor.rs
+148 -57
View File
@@ -35,11 +35,15 @@ use windmill_common::{
error::{self, to_anyhow, Result},
jobs::QueuedJob,
};
#[cfg(all(feature = "enterprise", feature = "parquet"))]
use windmill_common::s3_helpers::attempt_fetch_bytes;
use windmill_parser::Typ;
const RELATIVE_BUN_LOADER: &str = include_str!("../loader.bun.ts");
const RELATIVE_BUN_LOADER: &str = include_str!("../loader.bun.js");
const RELATIVE_BUN_BUILDER: &str = include_str!("../loader_builder.bun.ts");
const RELATIVE_BUN_BUILDER: &str = include_str!("../loader_builder.bun.js");
const NSJAIL_CONFIG_RUN_BUN_CONTENT: &str = include_str!("../nsjail/run.bun.config.proto");
@@ -76,7 +80,7 @@ pub async fn gen_lockfile(
} else {
let _ = write_file(
&job_dir,
"build.ts",
"build.js",
&format!(
r#"
{}
@@ -100,7 +104,7 @@ pub async fn gen_lockfile(
.current_dir(job_dir)
.env_clear()
.envs(common_bun_proc_envs.clone())
.args(vec!["run", "build.ts"])
.args(vec!["run", "build.js"])
.stdout(Stdio::piped())
.stderr(Stdio::piped());
let child_process = start_child_process(child_cmd, &*BUN_PATH).await?;
@@ -370,7 +374,7 @@ try {{
}}
const bo = await Bun.build({{
entrypoints: ["{job_dir}/wrapper.ts"],
entrypoints: ["{job_dir}/wrapper.mjs"],
outdir: "./",
target: "node",
plugins: [p],
@@ -389,7 +393,7 @@ if (!bo.success) {{
} else {
write_file(
&job_dir,
"loader.bun.ts",
"loader.bun.js",
&format!(
r#"
import {{ plugin }} from "bun";
@@ -450,9 +454,46 @@ pub async fn generate_wrapper_mjs(
Ok(())
}
#[cfg(all(feature = "enterprise", feature = "parquet"))]
pub async fn pull_codebase(w_id: &str, id: &str, job_dir: &str) -> Result<()> {
let path = windmill_common::s3_helpers::bundle(&w_id, &id);
let bun_cache_path = format!("{}/{}", BUN_CACHE_DIR, path);
let dst = format!("{job_dir}/main.js");
let dirs_splitted = bun_cache_path.split("/").collect_vec();
tokio::fs::create_dir_all(dirs_splitted[..dirs_splitted.len() - 1].join("/")).await?;
if tokio::fs::metadata(&bun_cache_path).await.is_ok() {
tracing::info!("loading {bun_cache_path} from cache");
tokio::fs::symlink(&bun_cache_path, dst).await?;
} else if let Some(os) = windmill_common::s3_helpers::OBJECT_STORE_CACHE_SETTINGS
.read()
.await
.clone()
{
let bytes = attempt_fetch_bytes(os, &path).await?;
if *windmill_common::worker::CLOUD_HOSTED {
tokio::fs::write(dst, &bytes).await?;
} else {
tokio::fs::write(&bun_cache_path, &bytes).await?;
tokio::fs::symlink(bun_cache_path, dst).await?;
}
// extract_tar(bytes, job_dir).await?;
}
return Ok(());
}
#[cfg(not(all(feature = "enterprise", feature = "parquet")))]
pub async fn pull_codebase(_w_id: &str, _id: &str, _job_dir: &str) -> Result<()> {
return Err(error::Error::ExecutionErr(
"codebase is an EE feature".to_string(),
));
}
#[tracing::instrument(level = "trace", skip_all)]
pub async fn handle_bun_job(
requirements_o: Option<String>,
codebase: Option<String>,
mem_peak: &mut i32,
canceled_by: &mut Option<CanceledBy>,
job: &QueuedJob,
@@ -465,13 +506,20 @@ pub async fn handle_bun_job(
envs: HashMap<String, String>,
shared_mount: &str,
) -> error::Result<Box<RawValue>> {
let _ = write_file(job_dir, "main.ts", inner_content).await?;
if !codebase.is_some() {
let _ = write_file(job_dir, "main.ts", inner_content).await?;
} else {
let _ = write_file(job_dir, "package.json", r#"{ "type": "module" }"#).await?;
}
let common_bun_proc_envs: HashMap<String, String> =
get_common_bun_proc_envs(&base_internal_url).await;
let annotation = get_annotation(inner_content);
let mut annotation = get_annotation(inner_content);
if codebase.is_some() {
annotation.nodejs_mode = true
}
let main_override = get_main_override(job.args.as_ref());
#[cfg(not(feature = "enterprise"))]
@@ -481,7 +529,9 @@ pub async fn handle_bun_job(
));
}
if let Some(reqs) = requirements_o {
if let Some(codebase) = codebase.as_ref() {
pull_codebase(&job.workspace_id, codebase, job_dir).await?;
} else if let Some(reqs) = requirements_o {
let splitted = reqs.split(BUN_LOCKB_SPLIT).collect::<Vec<&str>>();
if splitted.len() != 2 {
return Err(error::Error::ExecutionErr(
@@ -547,10 +597,11 @@ pub async fn handle_bun_job(
// }
}
let main_code = remove_pinned_imports(inner_content)?;
let _ = write_file(job_dir, "main.ts", &main_code).await?;
let _ = write_file(job_dir, "main.ts", &remove_pinned_imports(inner_content)?).await?;
let init_logs = if annotation.nodejs_mode {
let init_logs = if codebase.is_some() {
"\n\n--- NODE SNAPSHOT EXECUTION ---\n".to_string()
} else if annotation.nodejs_mode {
"\n\n--- NODE CODE EXECUTION ---\n".to_string()
} else {
"\n\n--- BUN CODE EXECUTION ---\n".to_string()
@@ -581,11 +632,17 @@ pub async fn handle_bun_job(
// we cannot use Bun.read and Bun.write because it results in an EBADF error on cloud
let main_name = main_override.unwrap_or("main".to_string());
let main_import = if codebase.is_some() {
"./main.js"
} else {
"./main.ts"
};
let wrapper_content: String = format!(
r#"
import {{ {main_name} }} from "./main.ts";
import * as Main from "{main_import}";
const fs = require('fs/promises');
import * as fs from "fs/promises";
const args = await fs.readFile('args.json', {{ encoding: 'utf8' }}).then(JSON.parse)
.then(({{ {spread} }}) => [ {spread} ])
@@ -596,7 +653,7 @@ BigInt.prototype.toJSON = function () {{
{dates}
async function run() {{
let res: any = await {main_name}(...args);
let res = await Main.{main_name}(...args);
const res_json = JSON.stringify(res ?? null, (key, value) => typeof value === 'undefined' ? null : value);
await fs.writeFile("result.json", res_json);
process.exit(0);
@@ -614,7 +671,7 @@ try {{
}}
"#,
);
write_file(job_dir, "wrapper.ts", &wrapper_content).await?;
write_file(job_dir, "wrapper.mjs", &wrapper_content).await?;
Ok(()) as error::Result<()>
};
@@ -633,15 +690,19 @@ try {{
};
let write_loader_f = async {
build_loader(
job_dir,
base_internal_url,
&client.get_token().await,
&job.workspace_id,
&job.script_path(),
annotation.nodejs_mode,
)
.await
if !codebase.is_some() {
build_loader(
job_dir,
base_internal_url,
&client.get_token().await,
&job.workspace_id,
&job.script_path(),
annotation.nodejs_mode,
)
.await
} else {
Ok(())
}
};
let (reserved_variables, _, _) = tokio::try_join!(
@@ -650,7 +711,7 @@ try {{
write_loader_f
)?;
if annotation.nodejs_mode {
if annotation.nodejs_mode && !codebase.is_some() {
generate_wrapper_mjs(
job_dir,
&job.workspace_id,
@@ -705,6 +766,15 @@ try {{
&NODE_PATH,
"/tmp/nodejs/wrapper.mjs",
]
} else if codebase.is_some() {
vec![
"--config",
"run.config.proto",
"--",
&BUN_PATH,
"run",
"/tmp/bun/wrapper.mjs",
]
} else {
vec![
"--config",
@@ -715,8 +785,8 @@ try {{
"-i",
"--prefer-offline",
"-r",
"/tmp/bun/loader.bun.ts",
"/tmp/bun/wrapper.ts",
"/tmp/bun/loader.bun.js",
"/tmp/bun/wrapper.mjs",
]
};
nsjail_cmd
@@ -746,23 +816,28 @@ try {{
.stderr(Stdio::piped());
bun_cmd
} else {
let script_path = format!("{job_dir}/wrapper.ts");
let script_path = format!("{job_dir}/wrapper.mjs");
let mut bun_cmd = Command::new(&*BUN_PATH);
let args = if codebase.is_some() {
vec!["run", &script_path]
} else {
vec![
"run",
"-i",
"--prefer-offline",
"-r",
"./loader.bun.js",
&script_path,
]
};
bun_cmd
.current_dir(job_dir)
.env_clear()
.envs(envs)
.envs(reserved_variables)
.envs(common_bun_proc_envs)
.args(vec![
"run",
"-i",
"--prefer-offline",
"-r",
"./loader.bun.ts",
&script_path,
])
.args(args)
.stdout(Stdio::piped())
.stderr(Stdio::piped());
bun_cmd
@@ -824,6 +899,7 @@ use std::sync::Arc;
#[cfg(feature = "enterprise")]
pub async fn start_worker(
requirements_o: Option<String>,
codebase: Option<String>,
db: &sqlx::Pool<sqlx::Postgres>,
inner_content: &str,
base_internal_url: &str,
@@ -840,7 +916,13 @@ pub async fn start_worker(
let mut logs = "".to_string();
let mut mem_peak: i32 = 0;
let mut canceled_by: Option<CanceledBy> = None;
let _ = write_file(job_dir, "main.ts", inner_content).await?;
tracing::info!("Starting worker {w_id};{script_path} (codebase: {codebase:?}");
if !codebase.is_some() {
let _ = write_file(job_dir, "main.ts", inner_content).await?;
} else {
let _ = write_file(job_dir, "package.json", r#"{ "type": "module" }"#).await?;
}
let common_bun_proc_envs: HashMap<String, String> =
get_common_bun_proc_envs(&base_internal_url).await;
@@ -867,8 +949,10 @@ pub async fn start_worker(
)
.await;
let context_envs = build_envs_map(context.to_vec()).await;
let annotation = get_annotation(inner_content);
if let Some(reqs) = requirements_o {
if let Some(codebase) = codebase.as_ref() {
pull_codebase(w_id, codebase, job_dir).await?;
} else if let Some(reqs) = requirements_o {
let splitted = reqs.split(BUN_LOCKB_SPLIT).collect::<Vec<&str>>();
if splitted.len() != 2 {
return Err(error::Error::ExecutionErr(
@@ -959,10 +1043,15 @@ pub async fn start_worker(
""
};
let main_import = if codebase.is_some() {
"./main.js"
} else {
"./main.ts"
};
let wrapper_content: String = format!(
r#"
import {{ main }} from "./main.ts";
import {{ createInterface }} from "node:readline"
import * as Main from "{main_import}";
import * as Readline from "node:readline"
BigInt.prototype.toJSON = function () {{
return this.toString();
@@ -972,7 +1061,7 @@ BigInt.prototype.toJSON = function () {{
console.log('start');
for await (const line of createInterface({{ input: process.stdin }})) {{
for await (const line of Readline.createInterface({{ input: process.stdin }})) {{
{print_lines}
if (line === "end") {{
@@ -980,7 +1069,7 @@ for await (const line of createInterface({{ input: process.stdin }})) {{
}}
try {{
let {{ {spread} }} = JSON.parse(line)
let res: any = await main(...[ {spread} ]);
let res = await Main.main(...[ {spread} ]);
console.log("wm_res[success]:" + JSON.stringify(res ?? null, (key, value) => typeof value === 'undefined' ? null : value));
}} catch (e) {{
console.log("wm_res[error]:" + JSON.stringify({{ message: e.message, name: e.name, stack: e.stack, line: line }}));
@@ -988,20 +1077,22 @@ for await (const line of createInterface({{ input: process.stdin }})) {{
}}
"#,
);
write_file(job_dir, "wrapper.ts", &wrapper_content).await?;
write_file(job_dir, "wrapper.mjs", &wrapper_content).await?;
}
build_loader(
job_dir,
base_internal_url,
token,
w_id,
script_path,
annotation.nodejs_mode,
)
.await?;
if !codebase.is_some() {
build_loader(
job_dir,
base_internal_url,
token,
w_id,
script_path,
annotation.nodejs_mode,
)
.await?;
}
if annotation.nodejs_mode {
if annotation.nodejs_mode && !codebase.is_some() {
generate_wrapper_mjs(
job_dir,
w_id,
@@ -1050,8 +1141,8 @@ for await (const line of createInterface({{ input: process.stdin }})) {{
"-i",
"--prefer-offline",
"-r",
"./loader.bun.ts",
&format!("{job_dir}/wrapper.ts"),
"./loader.bun.js",
&format!("{job_dir}/wrapper.mjs"),
],
killpill_rx,
job_completed_tx,
backend/windmill-worker/src/common.rs
+62 -35
View File
@@ -26,7 +26,7 @@ use windmill_common::s3_helpers::OBJECT_STORE_CACHE_SETTINGS;
use windmill_common::s3_helpers::{
get_etag_or_empty, LargeFileStorage, ObjectStoreResource, S3Object,
};
use windmill_common::worker::{CLOUD_HOSTED, WORKER_CONFIG};
use windmill_common::worker::{CLOUD_HOSTED, TMP_DIR, WORKER_CONFIG};
use windmill_common::{
error::{self, Error},
jobs::QueuedJob,
@@ -39,6 +39,7 @@ use windmill_queue::{append_logs, CanceledBy};
#[cfg(any(target_os = "linux", target_os = "macos"))]
use std::os::unix::process::ExitStatusExt;
use std::process::ExitStatus;
use std::sync::atomic::AtomicU32;
use std::sync::Arc;
use std::{
@@ -69,7 +70,7 @@ use futures::{
use crate::{
AuthedClient, AuthedClientBackgroundTask, JOB_DEFAULT_TIMEOUT, MAX_RESULT_SIZE,
MAX_TIMEOUT_DURATION, MAX_WAIT_FOR_SIGINT, MAX_WAIT_FOR_SIGTERM, ROOT_CACHE_DIR, TMP_DIR,
MAX_TIMEOUT_DURATION, MAX_WAIT_FOR_SIGINT, MAX_WAIT_FOR_SIGTERM, ROOT_CACHE_DIR,
};
pub async fn build_args_map<'a>(
@@ -482,7 +483,7 @@ pub async fn run_future_with_polling_update_job_poller<Fut, T>(
result_f: Fut,
worker_name: &str,
w_id: &str,
) -> anyhow::Result<T>
) -> error::Result<T>
where
Fut: Future<Output = anyhow::Result<T>>,
{
@@ -514,12 +515,22 @@ where
tracing::error!("Query timeout: {}", e);
Error::ExecutionErr(format!("Query timeout after (>{}s)", timeout_ms/1000))
})?,
_ = update_job, if job_id != Uuid::nil() => Err(Error::ExecutionErr("Job cancelled".to_string())).map_err(to_anyhow)?,
ex = update_job, if job_id != Uuid::nil() => {
match ex {
UpdateJobPollingExit::Done => Err(Error::ExecutionErr("Job cancelled".to_string())).map_err(to_anyhow)?,
UpdateJobPollingExit::AlreadyCompleted => Err(Error::AlreadyCompleted("Job already completed".to_string())).map_err(to_anyhow)?,
}
}
}?;
drop(tx);
Ok(rows)
}
pub enum UpdateJobPollingExit {
Done,
AlreadyCompleted,
}
pub async fn update_job_poller<F, Fut>(
job_id: Uuid,
db: &DB,
@@ -529,13 +540,14 @@ pub async fn update_job_poller<F, Fut>(
worker_name: &str,
w_id: &str,
mut rx: broadcast::Receiver<()>,
) where
) -> UpdateJobPollingExit
where
F: Fn() -> Fut,
Fut: Future<Output = i32>,
{
let update_job_interval = Duration::from_millis(500);
if job_id == Uuid::nil() {
return;
return UpdateJobPollingExit::Done;
}
let db = db.clone();
@@ -556,7 +568,9 @@ pub async fn update_job_poller<F, Fut>(
i+=1;
if i % 10 == 0 {
sqlx::query!(
"UPDATE worker_ping SET ping_at = now() WHERE worker = $1",
"UPDATE worker_ping SET ping_at = now(), current_job_id = $1, current_job_workspace_id = $2 WHERE worker = $3",
&job_id,
&w_id,
&worker_name
)
.execute(&db)
@@ -593,28 +607,36 @@ pub async fn update_job_poller<F, Fut>(
}
}
let (canceled, canceled_by, canceled_reason) = sqlx::query_as::<_, (bool, Option<String>, Option<String>)>("UPDATE queue SET mem_peak = $1, last_ping = now() WHERE id = $2 RETURNING canceled, canceled_by, canceled_reason")
let (canceled, canceled_by, canceled_reason, already_completed) = sqlx::query_as::<_, (bool, Option<String>, Option<String>, bool)>("UPDATE queue SET mem_peak = $1, last_ping = now() WHERE id = $2 RETURNING canceled, canceled_by, canceled_reason, false")
.bind(*mem_peak)
.bind(job_id)
.fetch_optional(&db)
.await
.unwrap_or_else(|e| {
tracing::error!(%e, "error updating job {job_id}: {e}");
Some((false, None, None))
Some((false, None, None, false))
})
.unwrap_or((false, None, None));
.unwrap_or_else(|| {
// if the job is not in queue, it can only be in the completed_job so it is already complete
(false, None, None, true)
});
if already_completed {
return UpdateJobPollingExit::AlreadyCompleted
}
if canceled {
canceled_by_ref.replace(CanceledBy {
username: canceled_by.clone(),
reason: canceled_reason.clone(),
});
break;
break
}
}
},
);
}
tracing::info!("job {job_id} finished");
UpdateJobPollingExit::Done
}
pub enum CompactLogs {
@@ -630,7 +652,7 @@ async fn compact_logs(
nlogs: String,
total_size: Arc<AtomicU32>,
compact_kind: CompactLogs,
worker_name: &str,
_worker_name: &str,
) -> error::Result<(String, String)> {
let size = sqlx::query_scalar!(
"SELECT char_length(logs) FROM job_logs WHERE job_id = $1 AND workspace_id = $2",
@@ -695,9 +717,9 @@ async fn compact_logs(
);
let mut new_current_logs = match compact_kind {
CompactLogs::NoS3 => format!("[windmill] worker {worker_name}: Logs length has exceeded a threshold\n[windmill] Previous logs have been saved to disk at {path}, add object storage in the instance settings to save it on distributed storage and allow direct download from Windmill\n"),
CompactLogs::NoS3 => format!("[windmill] No object storage set in instance settings. Previous logs have been saved to disk at {path}\n"),
CompactLogs::S3 => format!("[windmill] Previous logs have been saved to object storage at {path}\n"),
CompactLogs::NotEE => format!("[windmill] worker {worker_name}: Logs length has exceeded a threshold\n[windmill] Previous logs have been saved to disk at {path}\n[windmill] Upgrade to EE and add object storage to save it persistentely on distributed storage and allow direct download from Windmill\n"),
CompactLogs::NotEE => format!("[windmill] Previous logs have been saved to disk at {path}\n"),
};
new_current_logs.push_str(&current_logs);
@@ -897,6 +919,7 @@ pub async fn handle_child(
TooManyLogs,
Timeout,
Cancelled,
AlreadyCompleted,
}
let (timeout_duration, timeout_warn_msg) =
@@ -914,7 +937,10 @@ pub async fn handle_child(
result = child.wait() => return result.map(Ok),
Ok(()) = too_many_logs.changed() => KillReason::TooManyLogs,
_ = sleep(timeout_duration) => KillReason::Timeout,
_ = update_job, if job_id != Uuid::nil() => KillReason::Cancelled,
ex = update_job, if job_id != Uuid::nil() => match ex {
UpdateJobPollingExit::Done => KillReason::Cancelled,
UpdateJobPollingExit::AlreadyCompleted => KillReason::AlreadyCompleted,
},
};
tx.send(()).expect("rx should never be dropped");
drop(tx);
@@ -1120,26 +1146,7 @@ pub async fn handle_child(
_ if *too_many_logs.borrow() => Err(Error::ExecutionErr(format!(
"logs or result reached limit. (current max size: {MAX_RESULT_SIZE} characters)"
))),
Ok(Ok(status)) => {
if status.success() {
Ok(())
} else if let Some(code) = status.code() {
Err(error::Error::ExitStatus(code))
} else {
#[cfg(any(target_os = "linux", target_os = "macos"))]
return Err(error::Error::ExecutionErr(format!(
"process terminated by signal: {:#?}, stopped_signal: {:#?}, core_dumped: {}",
status.signal(),
status.stopped_signal(),
status.core_dumped()
)));
#[cfg(not(any(target_os = "linux", target_os = "macos")))]
return Err(error::Error::ExecutionErr(String::from(
"process terminated by signal",
)));
}
}
Ok(Ok(status)) => process_status(status),
Ok(Err(kill_reason)) => Err(Error::ExecutionErr(format!(
"job process killed because {kill_reason:#?}"
))),
@@ -1147,6 +1154,26 @@ pub async fn handle_child(
}
}
pub fn process_status(status: ExitStatus) -> error::Result<()> {
if status.success() {
Ok(())
} else if let Some(code) = status.code() {
Err(error::Error::ExitStatus(code))
} else {
#[cfg(any(target_os = "linux", target_os = "macos"))]
return Err(error::Error::ExecutionErr(format!(
"process terminated by signal: {:#?}, stopped_signal: {:#?}, core_dumped: {}",
status.signal(),
status.stopped_signal(),
status.core_dumped()
)));
#[cfg(not(any(target_os = "linux", target_os = "macos")))]
return Err(error::Error::ExecutionErr(String::from(
"process terminated by signal",
)));
}
}
pub async fn start_child_process(mut cmd: Command, executable: &str) -> Result<Child, Error> {
return cmd
.spawn()
backend/windmill-worker/src/dedicated_worker.rs
+7 -2
View File
@@ -16,7 +16,8 @@ use std::{collections::VecDeque, process::Stdio, sync::Arc};
use anyhow::Context;
use crate::{
common::start_child_process, JobCompleted, JobCompletedSender, MAX_BUFFERED_DEDICATED_JOBS,
common::{process_status, start_child_process},
JobCompleted, JobCompletedSender, MAX_BUFFERED_DEDICATED_JOBS,
};
use futures::{future, Future};
@@ -108,7 +109,11 @@ pub async fn handle_dedicated_process(
.wait()
.await
.expect("child process encountered an error");
tracing::info!("child status was: {}", status);
if let Err(e) = process_status(status) {
tracing::error!("child exit status was not success: {e}");
} else {
tracing::info!("child exist status was success");
}
});
let mut jobs = VecDeque::with_capacity(MAX_BUFFERED_DEDICATED_JOBS);
backend/windmill-worker/src/global_cache.rs
+6 -46
View File
@@ -71,50 +71,10 @@ pub async fn build_tar_and_push(
Ok(())
}
#[cfg(all(feature = "enterprise", feature = "parquet"))]
pub async fn attempt_fetch_bytes(
client: Arc<dyn ObjectStore>,
path: &str,
) -> error::Result<bytes::Bytes> {
use object_store::path::Path;
let object = client.get(&Path::from(path.clone())).await;
if let Err(e) = object {
tracing::info!(
"Failed to pull bytes from object store at path {path}. Error: {:?}",
e
);
return Err(error::Error::ExecutionErr(format!(
"Failed to pull bytes from object store: {path}"
)));
}
let bytes = object.unwrap().bytes().await;
if bytes.is_err() {
tracing::info!(
"Failed to read bytes from object store: {path}. Error: {:?}",
bytes.err()
);
return Err(error::Error::ExecutionErr(format!(
"Failed to read bytes from object store: {path}"
)));
}
let bytes = bytes.unwrap();
tracing::info!("{path} len: {}", bytes.len());
if bytes.len() == 0 {
tracing::info!("object {path} not found in bucket, bytes empty",);
return Err(error::Error::ExecutionErr(format!(
"object {path} does not exist in bucket"
)));
}
return Ok(bytes);
}
#[cfg(all(feature = "enterprise", feature = "parquet"))]
pub async fn pull_from_tar(client: Arc<dyn ObjectStore>, folder: String) -> error::Result<()> {
use windmill_common::s3_helpers::attempt_fetch_bytes;
let folder_name = folder.split("/").last().unwrap();
tracing::info!("Attempting to pull piptar {folder_name} from bucket");
@@ -125,7 +85,7 @@ pub async fn pull_from_tar(client: Arc<dyn ObjectStore>, folder: String) -> erro
// tracing::info!("B: {target} {folder}");
extract_pip_tar(bytes, &folder).await.map_err(|e| {
extract_tar(bytes, &folder).await.map_err(|e| {
tracing::error!("Failed to extract piptar {folder_name}. Error: {:?}", e);
e
})?;
@@ -139,7 +99,7 @@ pub async fn pull_from_tar(client: Arc<dyn ObjectStore>, folder: String) -> erro
}
#[cfg(all(feature = "enterprise", feature = "parquet"))]
pub async fn extract_pip_tar(tar: bytes::Bytes, folder: &str) -> error::Result<()> {
pub async fn extract_tar(tar: bytes::Bytes, folder: &str) -> error::Result<()> {
use bytes::Buf;
use tokio::fs::{self};
@@ -149,14 +109,14 @@ pub async fn extract_pip_tar(tar: bytes::Bytes, folder: &str) -> error::Result<(
let mut ar = tar::Archive::new(tar.reader());
if let Err(e) = ar.unpack(folder) {
tracing::info!("Failed to untar piptar. Error: {:?}", e);
tracing::info!("Failed to untar to {folder}. Error: {:?}", e);
fs::remove_dir_all(&folder).await?;
return Err(error::Error::ExecutionErr(format!(
"Failed to untar piptar {folder}"
)));
}
tracing::info!(
"Finished extracting pip tar {folder}. Took {}ms",
"Finished extracting tar to {folder}. Took {}ms",
start.elapsed().as_millis(),
);
Ok(())
backend/windmill-worker/src/go_executor.rs
+1 -1
View File
@@ -120,7 +120,7 @@ async fn load_cache(bin_path: &str, _hash: &str) -> (bool, String) {
.await
.clone()
{
use crate::global_cache::attempt_fetch_bytes;
use windmill_common::s3_helpers::attempt_fetch_bytes;
if let Ok(mut x) = attempt_fetch_bytes(os, &hash_to_os_path(_hash)).await {
if let Err(e) = write_binary_file(bin_path, &mut x).await {
backend/windmill-worker/src/js_eval.rs
+19 -3
View File
@@ -579,6 +579,7 @@ pub struct LogString {
}
pub async fn eval_fetch_timeout(
env_code: String,
ts_expr: String,
js_expr: String,
args: Option<&Json<HashMap<String, Box<RawValue>>>>,
@@ -681,7 +682,7 @@ pub async fn eval_fetch_timeout(
let future = async {
tokio::select! {
r = eval_fetch(&mut js_runtime, &js_expr) => Ok(r),
r = eval_fetch(&mut js_runtime, &js_expr, Some(env_code)) => Ok(r),
_ = memory_limit_rx.recv() => Err(Error::ExecutionErr("Memory limit reached, killing isolate".to_string()))
}
};
@@ -722,11 +723,26 @@ pub async fn eval_fetch_timeout(
Ok(r)
}
async fn eval_fetch(js_runtime: &mut JsRuntime, expr: &str) -> anyhow::Result<Box<RawValue>> {
const WINDMILL_CLIENT: &str = include_str!("./windmill-client.js");
async fn eval_fetch(
js_runtime: &mut JsRuntime,
expr: &str,
env_code: Option<String>,
) -> anyhow::Result<Box<RawValue>> {
if let Some(env_code) = env_code.as_ref() {
let _ = js_runtime
.load_side_module(
&deno_core::resolve_url("file:///windmill.ts")?,
Some(format!("{env_code}\n{}", WINDMILL_CLIENT.to_string()).into()),
)
.await?;
}
let _ = js_runtime
.load_side_module(
&deno_core::resolve_url("file:///eval.ts")?,
Some(expr.to_string().into()),
Some(format!("{}\n{expr}", env_code.unwrap_or_default()).into()),
)
.await?;
backend/windmill-worker/src/runtime.js
+4 -2
View File
@@ -36,6 +36,8 @@ globalThis.FileReader = fileReader.FileReader;
globalThis.console = new console.Console((msg, level) =>
globalThis.Deno.core.ops.op_log(msg)
);
globalThis.AbortController = abortSignal.AbortController;
globalThis.AbortSignal = abortSignal.AbortSignal;
// Object.assign(globalThis, {
// console: nonEnumerable(
// new console.Console((msg, level) => core.print(msg, level > 1))
@@ -122,8 +124,8 @@ globalThis.console = new console.Console((msg, level) =>
// FormData: nonEnumerable(formData.FormData),
// // abort signal
// AbortController: nonEnumerable(abortSignal.AbortController),
// AbortSignal: nonEnumerable(abortSignal.AbortSignal),
// AbortController: nonEnumerable(abortSignal.AbortController),
// AbortSignal: nonEnumerable(abortSignal.AbortSignal),
// // // web sockets
// // WebSocket: nonEnumerable(webSocket.WebSocket),
backend/windmill-worker/src/windmill-client.js
+10380
View File
File diff suppressed because it is too large Load Diff
backend/windmill-worker/src/worker.rs
+126 -57
View File
@@ -6,6 +6,8 @@
* LICENSE-AGPL for a copy of the license.
*/
use windmill_common::worker::TMP_DIR;
use anyhow::Result;
use const_format::concatcp;
#[cfg(feature = "prometheus")]
@@ -41,7 +43,7 @@ use windmill_common::{
flows::{FlowModule, FlowModuleValue, FlowValue},
get_latest_deployed_hash_for_path,
jobs::{JobKind, JobPayload, QueuedJob},
scripts::{get_full_hub_script_by_path, ScriptHash, ScriptLang},
scripts::{get_full_hub_script_by_path, ScriptHash, ScriptLang, PREVIEW_IS_CODEBASE_HASH},
users::{SUPERADMIN_NOTIFICATION_EMAIL, SUPERADMIN_SECRET_EMAIL, SUPERADMIN_SYNC_EMAIL},
utils::{rd_string, StripPath},
worker::{
@@ -185,10 +187,9 @@ pub async fn create_token_for_owner(
Ok(token)
}
pub const TMP_DIR: &str = "/tmp/windmill";
pub const TMP_LOGS_DIR: &str = "/tmp/windmill/logs";
pub const TMP_LOGS_DIR: &str = concatcp!(TMP_DIR, "/logs");
pub const ROOT_CACHE_DIR: &str = "/tmp/windmill/cache/";
pub const ROOT_CACHE_DIR: &str = concatcp!(TMP_DIR, "/cache/");
pub const LOCK_CACHE_DIR: &str = concatcp!(ROOT_CACHE_DIR, "lock");
pub const PIP_CACHE_DIR: &str = concatcp!(ROOT_CACHE_DIR, "pip");
pub const TAR_PIP_CACHE_DIR: &str = concatcp!(ROOT_CACHE_DIR, "tar/pip");
@@ -198,6 +199,7 @@ pub const DENO_CACHE_DIR_NPM: &str = concatcp!(ROOT_CACHE_DIR, "deno/npm");
pub const GO_CACHE_DIR: &str = concatcp!(ROOT_CACHE_DIR, "go");
pub const BUN_CACHE_DIR: &str = concatcp!(ROOT_CACHE_DIR, "bun");
pub const HUB_CACHE_DIR: &str = concatcp!(ROOT_CACHE_DIR, "hub");
pub const GO_BIN_CACHE_DIR: &str = concatcp!(ROOT_CACHE_DIR, "gobin");
pub const POWERSHELL_CACHE_DIR: &str = concatcp!(ROOT_CACHE_DIR, "powershell");
@@ -606,7 +608,6 @@ pub async fn run_worker<R: rsmq_async::RsmqConnection + Send + Sync + Clone + 's
);
}
#[cfg(feature = "prometheus")]
let start_time = Instant::now();
let worker_dir = format!("{TMP_DIR}/{worker_name}");
@@ -918,6 +919,8 @@ pub async fn run_worker<R: rsmq_async::RsmqConnection + Send + Sync + Clone + 's
None
};
let mut worker_code_execution_metric: f32 = 0.0;
let mut jobs_executed = 0;
#[cfg(feature = "prometheus")]
@@ -1038,7 +1041,7 @@ pub async fn run_worker<R: rsmq_async::RsmqConnection + Send + Sync + Clone + 's
let same_worker_tx2 = same_worker_tx2.clone();
let rsmq2 = rsmq2.clone();
let worker_name = worker_name2.clone();
if matches!(jc.job.job_kind, JobKind::Noop) || is_dedicated_worker {
if matches!(jc.job.job_kind, JobKind::Noop) {
thread_count.fetch_add(1, Ordering::SeqCst);
let thread_count = thread_count.clone();
@@ -1130,15 +1133,19 @@ pub async fn run_worker<R: rsmq_async::RsmqConnection + Send + Sync + Clone + 's
.await
.expect("update config to trigger restart of all dedicated workers at that config");
killpill_tx.send(()).unwrap_or_default();
}
});
} else {
let is_init_script_and_failure =
!jc.success && jc.job.tag.as_str() == INIT_SCRIPT_TAG;
let is_dependency_job = matches!(
jc.job.job_kind,
JobKind::Dependencies | JobKind::FlowDependencies);
handle_receive_completed_job(
jc,
base_internal_url2,
db2,
db2.clone(),
worker_dir2,
same_worker_tx2,
rsmq2,
@@ -1152,6 +1159,15 @@ pub async fn run_worker<R: rsmq_async::RsmqConnection + Send + Sync + Clone + 's
tracing::error!("init script errored, exiting");
killpill_tx2.send(()).unwrap_or_default();
}
if is_dependency_job && is_dedicated_worker {
tracing::error!("Dedicated worker executed a dependency job, a new script has been deployed. Exiting expecting to be restarted.");
sqlx::query!("UPDATE config SET config = config WHERE name = $1", format!("worker__{}", *WORKER_GROUP))
.execute(&db2)
.await
.expect("update config to trigger restart of all dedicated workers at that config");
killpill_tx2.send(()).unwrap_or_default();
}
}
}
SendResult::UpdateFlow {
@@ -1373,14 +1389,12 @@ pub async fn run_worker<R: rsmq_async::RsmqConnection + Send + Sync + Clone + 's
let tags = WORKER_CONFIG.read().await.worker_tags.clone();
if let Err(e) = sqlx::query!(
"UPDATE worker_ping SET ping_at = now(), jobs_executed = $1, custom_tags = $2 WHERE worker = $3",
"UPDATE worker_ping SET ping_at = now(), jobs_executed = $1, custom_tags = $2, occupancy_rate = $3, current_job_id = NULL, current_job_workspace_id = NULL WHERE worker = $4",
jobs_executed,
tags.as_slice(),
worker_code_execution_metric / start_time.elapsed().as_secs_f32(),
&worker_name
)
.execute(db)
.await
{
).execute(db).await {
tracing::error!("failed to update worker ping, exiting: {}", e);
killpill_tx.send(()).unwrap_or_default();
}
@@ -1671,7 +1685,7 @@ pub async fn run_worker<R: rsmq_async::RsmqConnection + Send + Sync + Clone + 's
let tag = job.tag.clone();
let arc_job = Arc::new(job);
if let Some(err) = handle_queued_job(
if let Err(err) = handle_queued_job(
arc_job.clone(),
db,
&authed_client,
@@ -1682,11 +1696,11 @@ pub async fn run_worker<R: rsmq_async::RsmqConnection + Send + Sync + Clone + 's
base_internal_url,
rsmq.clone(),
job_completed_tx.clone(),
&mut worker_code_execution_metric,
worker_flow_initial_transition_duration.clone(),
worker_code_execution_duration.clone(),
)
.await
.err()
{
let is_init_script = arc_job.tag.as_str() == INIT_SCRIPT_TAG;
handle_job_error(
@@ -2026,18 +2040,21 @@ async fn spawn_dedicated_worker(
SpawnWorker::RawScript { path, .. } => path.to_string(),
SpawnWorker::Script { path, .. } => path.to_string(),
};
let path2 = path.clone();
let w_id = w_id.to_string();
let (content, lock, language, envs) = match sw {
let (content, lock, language, envs, codebase) = match sw {
SpawnWorker::Script { path, hash } => {
let q = if let Some(hash) = hash {
get_script_content_by_hash(&hash, &w_id, &db).await.map(
|r: ContentReqLangEnvs| Some((r.content, r.lockfile, r.language, r.envs)),
|r: ContentReqLangEnvs| {
Some((r.content, r.lockfile, r.language, r.envs, r.codebase))
},
)
} else {
sqlx::query_as::<_, (String, Option<String>, Option<ScriptLang>, Option<Vec<String>>)>(
"SELECT content, lock, language, envs FROM script WHERE path = $1 AND workspace_id = $2 AND
sqlx::query_as::<_, (String, Option<String>, Option<ScriptLang>, Option<Vec<String>>, bool, Option<ScriptHash>)>(
"SELECT content, lock, language, envs, codebase IS NOT NULL, hash FROM script WHERE path = $1 AND workspace_id = $2 AND
created_at = (SELECT max(created_at) FROM script WHERE path = $1 AND workspace_id = $2 AND
deleted = false AND lock IS not NULL AND lock_error_logs IS NULL)",
)
@@ -2046,6 +2063,7 @@ async fn spawn_dedicated_worker(
.fetch_optional(&db)
.await
.map_err(|e| Error::InternalErr(format!("expected content and lock: {e}")))
.map(|x| x.map(|y| (y.0, y.1, y.2, y.3, if y.4 { y.5.map(|z| z.to_string()) } else { None })))
};
if let Ok(q) = q {
if let Some(wp) = q {
@@ -2064,7 +2082,9 @@ async fn spawn_dedicated_worker(
return None;
}
}
SpawnWorker::RawScript { content, lock, lang, .. } => (content, lock, Some(lang), None),
SpawnWorker::RawScript { content, lock, lang, .. } => {
(content, lock, Some(lang), None, None)
}
};
match language {
@@ -2119,6 +2139,7 @@ async fn spawn_dedicated_worker(
Some(ScriptLang::Bun) => {
crate::bun_executor::start_worker(
lock,
codebase,
&db,
&content,
&base_internal_url,
@@ -2177,6 +2198,7 @@ async fn queue_init_bash_maybe<'c, R: rsmq_async::RsmqConnection + Send + 'c>(
tx,
"admins",
windmill_common::jobs::JobPayload::Code(windmill_common::jobs::RawCode {
hash: None,
content: content.clone(),
path: Some(format!("init_script_{worker_name}")),
language: ScriptLang::Bash,
@@ -2535,6 +2557,7 @@ pub struct JobCompleted {
async fn do_nativets(
job: &QueuedJob,
client: &AuthedClientBackgroundTask,
env_code: String,
code: String,
db: &Pool<Postgres>,
mem_peak: &mut i32,
@@ -2549,6 +2572,7 @@ async fn do_nativets(
};
let result = eval_fetch_timeout(
env_code,
code.clone(),
transpile_ts(code)?,
job_args,
@@ -2582,6 +2606,7 @@ async fn handle_queued_job<R: rsmq_async::RsmqConnection + Send + Sync + Clone>(
base_internal_url: &str,
rsmq: Option<R>,
job_completed_tx: JobCompletedSender,
worker_code_execution_metric: &mut f32,
_worker_flow_initial_transition_duration: Option<Histo>,
_worker_code_execution_duration: Option<Histo>,
) -> windmill_common::error::Result<()> {
@@ -2808,6 +2833,7 @@ async fn handle_queued_job<R: rsmq_async::RsmqConnection + Send + Sync + Clone>(
_ => {
#[cfg(feature = "prometheus")]
let timer = _worker_code_execution_duration.map(|x| x.start_timer());
let metric_timer = Instant::now();
let r = handle_code_execution_job(
job.as_ref(),
db,
@@ -2821,6 +2847,7 @@ async fn handle_queued_job<R: rsmq_async::RsmqConnection + Send + Sync + Clone>(
&mut column_order,
)
.await;
*worker_code_execution_metric += metric_timer.elapsed().as_secs_f32();
#[cfg(feature = "prometheus")]
timer.map(|x| x.stop_and_record());
r
@@ -2831,6 +2858,14 @@ async fn handle_queued_job<R: rsmq_async::RsmqConnection + Send + Sync + Clone>(
if job.as_ref().workspace_id == "" {
return Ok(());
}
if result
.as_ref()
.is_err_and(|err| matches!(err, &Error::AlreadyCompleted(_)))
{
return Ok(());
}
process_result(
job,
result,
@@ -2975,6 +3010,7 @@ struct ContentReqLangEnvs {
lockfile: Option<String>,
language: Option<ScriptLang>,
envs: Option<Vec<String>>,
codebase: Option<String>,
}
async fn get_hub_script_content_and_requirements(
@@ -3011,6 +3047,7 @@ async fn get_hub_script_content_and_requirements(
lockfile: script.lockfile,
language: Some(script.language),
envs: None,
codebase: None,
})
}
@@ -3043,16 +3080,27 @@ async fn get_script_content_by_hash(
Option<String>,
Option<ScriptLang>,
Option<Vec<String>>,
bool,
),
>(
"SELECT content, lock, language, envs FROM script WHERE hash = $1 AND workspace_id = $2",
"SELECT content, lock, language, envs, codebase IS NOT NULL FROM script WHERE hash = $1 AND workspace_id = $2",
)
.bind(script_hash.0)
.bind(w_id)
.fetch_optional(db)
.await?
.ok_or_else(|| Error::InternalErr(format!("expected content and lock")))?;
Ok(ContentReqLangEnvs { content: r.0, lockfile: r.1, language: r.2, envs: r.3 })
Ok(ContentReqLangEnvs {
content: r.0,
lockfile: r.1,
language: r.2,
envs: r.3,
codebase: if r.4 {
Some(script_hash.to_string())
} else {
None
},
})
}
#[tracing::instrument(level = "trace", skip_all)]
@@ -3068,35 +3116,48 @@ async fn handle_code_execution_job(
worker_name: &str,
column_order: &mut Option<Vec<String>>,
) -> error::Result<Box<RawValue>> {
let ContentReqLangEnvs { content: inner_content, lockfile: requirements_o, language, envs } =
match job.job_kind {
JobKind::Preview => ContentReqLangEnvs {
content: job
.raw_code
.clone()
.unwrap_or_else(|| "no raw code".to_owned()),
lockfile: job.raw_lock.clone(),
language: job.language.to_owned(),
envs: None,
let ContentReqLangEnvs {
content: inner_content,
lockfile: requirements_o,
language,
envs,
codebase,
} = match job.job_kind {
JobKind::Preview => ContentReqLangEnvs {
content: job
.raw_code
.clone()
.unwrap_or_else(|| "no raw code".to_owned()),
lockfile: job.raw_lock.clone(),
language: job.language.to_owned(),
envs: None,
codebase: if job
.script_hash
.is_some_and(|y| y.0 == PREVIEW_IS_CODEBASE_HASH)
{
Some(job.id.to_string())
} else {
None
},
JobKind::Script_Hub => {
get_hub_script_content_and_requirements(job.script_path.clone(), db).await?
}
JobKind::Script => {
get_script_content_by_hash(
&job.script_hash.unwrap_or(ScriptHash(0)),
&job.workspace_id,
db,
)
.await?
}
JobKind::DeploymentCallback => {
get_script_content_by_path(job.script_path.clone(), &job.workspace_id, db).await?
}
_ => unreachable!(
"handle_code_execution_job should never be reachable with a non-code execution job"
),
};
},
JobKind::Script_Hub => {
get_hub_script_content_and_requirements(job.script_path.clone(), db).await?
}
JobKind::Script => {
get_script_content_by_hash(
&job.script_hash.unwrap_or(ScriptHash(0)),
&job.workspace_id,
db,
)
.await?
}
JobKind::DeploymentCallback => {
get_script_content_by_path(job.script_path.clone(), &job.workspace_id, db).await?
}
_ => unreachable!(
"handle_code_execution_job should never be reachable with a non-code execution job"
),
};
if language == Some(ScriptLang::Postgresql) {
return do_postgresql(
@@ -3209,17 +3270,24 @@ async fn handle_code_execution_job(
let reserved_variables = get_reserved_variables(job, &client.get_token().await, db).await?;
let code = format!(
"const BASE_URL = '{base_internal_url}';\nconst BASE_INTERNAL_URL = '{base_internal_url}';\n{}\n{}",
let env_code = format!(
"const process = {{ env: {{}} }};\nconst BASE_URL = '{base_internal_url}';\nconst BASE_INTERNAL_URL = '{base_internal_url}';\nprocess.env['BASE_URL'] = BASE_URL;process.env['BASE_INTERNAL_URL'] = BASE_INTERNAL_URL;\n{}",
reserved_variables
.iter()
.map(|(k, v)| format!("const {} = '{}';\n", k, v))
.map(|(k, v)| format!("const {} = '{}';\nprocess.env['{}'] = '{}';\n", k, v, k, v))
.collect::<Vec<String>>()
.join("\n"),
inner_content
);
let (result, ts_logs) =
do_nativets(job, &client, code, db, mem_peak, canceled_by, worker_name).await?;
.join("\n"));
let (result, ts_logs) = do_nativets(
job,
&client,
env_code,
inner_content,
db,
mem_peak,
canceled_by,
worker_name,
)
.await?;
append_logs(job.id, job.workspace_id.clone(), ts_logs, db).await;
return Ok(result);
}
@@ -3299,6 +3367,7 @@ mount {{
Some(ScriptLang::Bun) => {
handle_bun_job(
requirements_o,
codebase,
mem_peak,
canceled_by,
job,
backend/windmill-worker/src/worker_flow.rs
+1
View File
@@ -3124,6 +3124,7 @@ fn raw_script_to_payload(
) -> JobPayloadWithTag {
JobPayloadWithTag {
payload: JobPayload::Code(RawCode {
hash: None,
path,
content: content.clone(),
language: language.clone(),
benchmarks/lib.ts
+1 -1
View File
@@ -2,7 +2,7 @@ import { sleep } from "https://deno.land/x/sleep@v1.2.1/mod.ts";
import * as windmill from "https://deno.land/x/windmill@v1.174.0/mod.ts";
import * as api from "https://deno.land/x/windmill@v1.174.0/windmill-api/index.ts";
export const VERSION = "v1.323.6";
export const VERSION = "v1.327.0";
export async function login(email: string, password: string): Promise<string> {
return await windmill.UserService.login({
cli/codebase.ts
+21
View File
@@ -0,0 +1,21 @@
import { Codebase, SyncOptions } from "./conf.ts";
import { log } from "./deps.ts";
import { digestDir } from "./utils.ts";
export type SyncCodebase = Codebase & { digest: string };
export async function listSyncCodebases(
options: SyncOptions
): Promise<SyncCodebase[]> {
const res: SyncCodebase[] = [];
const nb_codebase = options?.codebases?.length ?? 0;
if (nb_codebase > 0) {
log.info(`Found ${nb_codebase} codebases:`);
}
for (const codebase of options?.codebases ?? []) {
const digest = await digestDir(codebase.relative_path);
log.info(`Codebase ${codebase.relative_path}, digest: ${digest}`);
res.push({ ...codebase, digest });
}
return res;
}
cli/conf.ts
+7
View File
@@ -20,6 +20,13 @@ export interface SyncOptions {
extraIncludes?: string[];
excludes?: string[];
defaultTs?: "bun" | "deno";
codebases?: Codebase[];
}
export interface Codebase {
relative_path: string;
includes?: string[];
excludes?: string[];
}
export async function readConfigFile(): Promise<SyncOptions> {
cli/flow.ts
+2
View File
@@ -71,6 +71,8 @@ export async function pushFlow(
}
} else if (m.value.type == "forloopflow") {
replaceInlineScripts(m.value.modules);
} else if (m.value.type == "whileloopflow") {
replaceInlineScripts(m.value.modules);
} else if (m.value.type == "branchall") {
m.value.branches.forEach((b) => replaceInlineScripts(b.modules));
} else if (m.value.type == "branchone") {
cli/main.ts
+21 -1
View File
@@ -3,7 +3,9 @@ import {
CompletionsCommand,
DenoLandProvider,
UpgradeCommand,
colors,
log,
yamlStringify,
} from "./deps.ts";
import flow from "./flow.ts";
import app from "./apps.ts";
@@ -31,7 +33,7 @@ addEventListener("error", (event) => {
}
});
export const VERSION = "v1.323.6";
export const VERSION = "v1.327.0";
let command: any = new Command()
.name("wmill")
@@ -58,6 +60,23 @@ let command: any = new Command()
"Specify headers to use for all requests. e.g: \"HEADERS='h1: v1, h2: v2'\""
)
.version(VERSION)
.command("init", "Bootstrap a windmill project with a wmill.yaml file")
.action(async () => {
if (await Deno.stat("wmill.yaml").catch(() => null)) {
log.error(colors.red("wmill.yaml already exists"));
return;
}
await Deno.writeTextFile(
"wmill.yaml",
yamlStringify({
defaultTs: "bun",
includes: [],
excludes: [],
codebases: [],
})
);
log.info(colors.green("wmill.yaml created"));
})
.command("app", app)
.command("flow", flow)
.command("script", script)
@@ -70,6 +89,7 @@ let command: any = new Command()
.command("schedule", schedule)
.command("dev", dev)
.command("sync", sync)
.command("version", "Show version information")
.action(async (opts) => {
console.log("CLI build against " + VERSION);
cli/metadata.ts
+7 -9
View File
@@ -1,6 +1,6 @@
// deno-lint-ignore-file no-explicit-any
import { GlobalOptions } from "./types.ts";
import { colors, encodeHex, log, yamlParse, yamlStringify } from "./deps.ts";
import { colors, log, yamlParse, yamlStringify } from "./deps.ts";
import {
ScriptMetadata,
defaultScriptMetadata,
@@ -25,6 +25,7 @@ import { ScriptLanguage } from "./script_common.ts";
import { inferContentTypeFromFilePath } from "./script_common.ts";
import { GlobalDeps } from "./script.ts";
import { yamlOptions } from "./sync.ts";
import { generateHash } from "./utils.ts";
export async function generateAllMetadata() {}
@@ -457,7 +458,11 @@ export async function replaceLock(o?: { lock?: string | string[] }) {
export async function parseMetadataFile(
scriptPath: string,
generateMetadataIfMissing:
| (GlobalOptions & { path: string; workspaceRemote: Workspace })
| (GlobalOptions & {
path: string;
workspaceRemote: Workspace;
schemaOnly?: boolean;
})
| undefined,
globalDeps: GlobalDeps
): Promise<{ isJson: boolean; payload: any; path: string }> {
@@ -509,7 +514,6 @@ export async function parseMetadataFile(
false,
globalDeps
);
scriptInitialMetadata = yamlParse(
await Deno.readTextFile(metadataFilePath)
) as ScriptMetadata;
@@ -552,12 +556,6 @@ export async function readLockfile(): Promise<Lock> {
}
}
async function generateHash(content: string): Promise<string> {
const messageBuffer = new TextEncoder().encode(content);
const hashBuffer = await crypto.subtle.digest("SHA-256", messageBuffer);
return encodeHex(hashBuffer);
}
export async function checkifMetadataUptodate(
path: string,
requirement: string
cli/script.ts
+153 -66
View File
@@ -7,7 +7,6 @@ import {
Confirm,
JobService,
log,
NewScript,
readAll,
Script,
ScriptService,
@@ -29,6 +28,7 @@ import {
} from "./script_common.ts";
import {
elementsToMap,
findCodebase,
readDirRecursiveWithIgnore,
yamlOptions,
} from "./sync.ts";
@@ -39,6 +39,7 @@ import {
mergeConfigWithConfigFile,
readConfigFile,
} from "./conf.ts";
import { SyncCodebase, listSyncCodebases } from "./codebase.ts";
export interface ScriptFile {
parent_hash?: string;
@@ -71,8 +72,18 @@ async function push(opts: PushOptions, filePath: string) {
}
await requireLogin(opts);
const globalDeps = await findGlobalDeps();
await handleFile(filePath, workspace, [], undefined, opts, globalDeps);
const codebases = await listSyncCodebases(opts as SyncOptions);
const globalDeps = await findGlobalDeps(codebases);
await handleFile(
filePath,
workspace,
[],
undefined,
opts,
globalDeps,
codebases
);
log.info(colors.bold.underline.green(`Script ${filePath} pushed`));
}
@@ -81,7 +92,9 @@ export async function handleScriptMetadata(
workspace: Workspace,
alreadySynced: string[],
message: string | undefined,
globalDeps: GlobalDeps
globalDeps: GlobalDeps,
codebases: SyncCodebase[],
opts: GlobalOptions
): Promise<boolean> {
if (
path.endsWith(".script.json") ||
@@ -94,8 +107,9 @@ export async function handleScriptMetadata(
workspace,
alreadySynced,
message,
undefined,
globalDeps
opts,
globalDeps,
codebases
);
} else {
return false;
@@ -108,7 +122,8 @@ export async function handleFile(
alreadySynced: string[],
message: string | undefined,
opts: (GlobalOptions & { defaultTs?: "bun" | "deno" }) | undefined,
globalDeps: GlobalDeps
globalDeps: GlobalDeps,
codebases: SyncCodebase[]
): Promise<boolean> {
if (
!path.includes(".inline_script.") &&
@@ -123,14 +138,40 @@ export async function handleFile(
const remotePath = path
.substring(0, path.indexOf("."))
.replaceAll("\\", "/");
const language = inferContentTypeFromFilePath(path, opts?.defaultTs);
const codebase =
language == "bun" ? findCodebase(path, codebases) : undefined;
let bundleContent: string | undefined = undefined;
if (codebase) {
const esbuild = await import("npm:esbuild");
log.info(`Starting building the bundle for ${path}`);
const out = await esbuild.build({
entryPoints: [path],
format: "esm",
bundle: true,
write: false,
});
bundleContent = out.outputFiles[0].text;
log.info(`Finished building the bundle for ${path}`);
}
const typed = (
await parseMetadataFile(
remotePath,
opts ? { ...opts, path, workspaceRemote: workspace } : undefined,
opts
? {
...opts,
path,
workspaceRemote: workspace,
schemaOnly: codebase ? true : undefined,
}
: undefined,
globalDeps
)
)?.payload;
const language = inferContentTypeFromFilePath(path, opts?.defaultTs);
const workspaceId = workspace.workspaceId;
@@ -146,6 +187,30 @@ export async function handleFile(
}
const content = await Deno.readTextFile(path);
const requestBodyCommon = {
content,
description: typed?.description ?? "",
language: language,
path: remotePath.replaceAll("\\", "/"),
summary: typed?.summary ?? "",
kind: typed?.kind,
lock: typed?.lock,
schema: typed?.schema,
tag: typed?.tag,
ws_error_handler_muted: typed?.ws_error_handler_muted,
dedicated_worker: typed?.dedicated_worker,
cache_ttl: typed?.cache_ttl,
concurrency_time_window_s: typed?.concurrency_time_window_s,
concurrent_limit: typed?.concurrent_limit,
deployment_message: message,
restart_unless_cancelled: typed?.restart_unless_cancelled,
visible_to_runner_only: typed?.visible_to_runner_only,
no_main_func: typed?.no_main_func,
priority: typed?.priority,
codebase: codebase?.digest,
timeout: typed?.timeout,
};
if (remote) {
if (content === remote.content) {
if (
@@ -172,7 +237,8 @@ export async function handleFile(
Boolean(typed.visible_to_runner_only) ==
Boolean(remote.visible_to_runner_only) &&
Boolean(typed.no_main_func) == Boolean(remote.no_main_func) &&
typed.priority == Boolean(remote.priority))
typed.priority == Boolean(remote.priority) &&
typed.timeout == remote.timeout)
) {
log.info(colors.green(`Script ${remotePath} is up to date`));
return true;
@@ -182,67 +248,85 @@ export async function handleFile(
log.info(
colors.yellow.bold(`Creating script with a parent ${remotePath}`)
);
await ScriptService.createScript({
workspace: workspaceId,
requestBody: {
content,
description: typed?.description ?? "",
language: language,
path: remotePath.replaceAll("\\", "/"),
summary: typed?.summary ?? "",
kind: typed?.kind,
lock: typed?.lock,
parent_hash: remote.hash,
schema: typed?.schema,
tag: typed?.tag,
ws_error_handler_muted: typed?.ws_error_handler_muted,
dedicated_worker: typed?.dedicated_worker,
cache_ttl: typed?.cache_ttl,
concurrency_time_window_s: typed?.concurrency_time_window_s,
concurrent_limit: typed?.concurrent_limit,
deployment_message: message,
restart_unless_cancelled: typed?.restart_unless_cancelled,
visible_to_runner_only: typed?.visible_to_runner_only,
no_main_func: typed?.no_main_func,
priority: typed?.priority,
},
});
const body = {
...requestBodyCommon,
parent_hash: remote.hash,
};
await createScript(bundleContent, workspaceId, body, workspace);
} else {
log.info(
colors.yellow.bold(`Creating script without parent ${remotePath}`)
);
// no parent hash
await ScriptService.createScript({
workspace: workspaceId,
requestBody: {
content,
description: typed?.description ?? "",
language: language,
path: remotePath.replaceAll("\\", "/"),
summary: typed?.summary ?? "",
kind: typed?.kind,
lock: typed?.lock,
parent_hash: undefined,
schema: typed?.schema,
tag: typed?.tag,
ws_error_handler_muted: typed?.ws_error_handler_muted,
dedicated_worker: typed?.dedicated_worker,
cache_ttl: typed?.cache_ttl,
concurrency_time_window_s: typed?.concurrency_time_window_s,
concurrent_limit: typed?.concurrent_limit,
deployment_message: message,
restart_unless_cancelled: typed?.restart_unless_cancelled,
visible_to_runner_only: typed?.visible_to_runner_only,
no_main_func: typed?.no_main_func,
priority: typed?.priority,
},
});
const body = {
...requestBodyCommon,
parent_hash: undefined,
};
await createScript(bundleContent, workspaceId, body, workspace);
}
return true;
}
return false;
}
async function createScript(
bundleContent: string | undefined,
workspaceId: string,
body: {
parent_hash: string | undefined;
content: string;
description: any;
language: ScriptLanguage;
path: string;
summary: any;
kind: any;
lock: any;
schema: any;
tag: any;
ws_error_handler_muted: any;
dedicated_worker: any;
cache_ttl: any;
concurrency_time_window_s: any;
concurrent_limit: any;
deployment_message: string | undefined;
restart_unless_cancelled: any;
visible_to_runner_only: any;
no_main_func: any;
priority: any;
},
workspace: Workspace
) {
if (!bundleContent) {
// no parent hash
await ScriptService.createScript({
workspace: workspaceId,
requestBody: body,
});
} else {
const form = new FormData();
form.append("script", JSON.stringify(body));
form.append("file", bundleContent);
const url =
workspace.remote +
"api/w/" +
workspace.workspaceId +
"/scripts/create_snapshot";
const req = await fetch(url, {
method: "POST",
headers: { Authorization: `Bearer ${workspace.token}` },
body: form,
});
if (req.status != 201) {
throw Error(
`Script snapshot creation was not successful: ${req.status} - ${
req.statusText
} - ${await req.text()}`
);
}
}
}
export async function findContentFile(filePath: string) {
const candidates = filePath.endsWith("script.json")
? exts.map((x) => filePath.replace(".script.json", x))
@@ -588,10 +672,12 @@ export type GlobalDeps = {
pkgs: Record<string, string>;
reqs: Record<string, string>;
};
export async function findGlobalDeps(): Promise<GlobalDeps> {
export async function findGlobalDeps(
codebases: SyncCodebase[]
): Promise<GlobalDeps> {
const pkgs: { [key: string]: string } = {};
const reqs: { [key: string]: string } = {};
const els = await FSFSElement(Deno.cwd());
const els = await FSFSElement(Deno.cwd(), codebases);
for await (const entry of readDirRecursiveWithIgnore((p, isDir) => {
p = "/" + p;
return (
@@ -614,7 +700,7 @@ async function generateMetadata(
schemaOnly?: boolean;
yes?: boolean;
} & SyncOptions,
scriptPath?: string
scriptPath: string | undefined
) {
if (scriptPath == "") {
scriptPath = undefined;
@@ -626,8 +712,9 @@ async function generateMetadata(
const workspace = await resolveWorkspace(opts);
await requireLogin(opts);
opts = await mergeConfigWithConfigFile(opts);
const codebases = await listSyncCodebases(opts);
const globalDeps = await findGlobalDeps();
const globalDeps = await findGlobalDeps(codebases);
if (scriptPath) {
// read script metadata file
await generateMetadataInternal(
@@ -641,7 +728,7 @@ async function generateMetadata(
} else {
const ignore = await ignoreF(opts);
const elems = await elementsToMap(
await FSFSElement(Deno.cwd()),
await FSFSElement(Deno.cwd(), codebases),
(p, isD) => {
return (
(!isD && !exts.some((ext) => p.endsWith(ext))) ||
cli/sync.ts
+103 -12
View File
@@ -45,6 +45,7 @@ import { handleFile } from "./script.ts";
import { deepEqual } from "./utils.ts";
import { SyncOptions, mergeConfigWithConfigFile } from "./conf.ts";
import { removePathPrefix } from "./types.ts";
import { SyncCodebase, listSyncCodebases } from "./codebase.ts";
type DynFSElement = {
isDirectory: boolean;
@@ -54,8 +55,85 @@ type DynFSElement = {
getChildren(): AsyncIterable<DynFSElement>;
};
export async function FSFSElement(p: string): Promise<DynFSElement> {
function _internal_element(localP: string, isDir: boolean): DynFSElement {
export function findCodebase(
path: string,
codebases: SyncCodebase[]
): SyncCodebase | undefined {
for (const c of codebases) {
let included = false;
let excluded = false;
if (c.includes == undefined || c.includes == null) {
included = true;
}
if (typeof c.includes == "string") {
c.includes = [c.includes];
}
for (const r of c.includes ?? []) {
if (included) {
break;
}
if (minimatch(path, r)) {
included = true;
}
}
if (typeof c.excludes == "string") {
c.excludes = [c.excludes];
}
for (const r of c.excludes ?? []) {
if (minimatch(path, r)) {
excluded = true;
}
}
if (included && !excluded) {
return c;
}
}
}
async function addCodebaseDigestIfRelevant(
path: string,
content: string,
codebases: SyncCodebase[]
): Promise<string> {
const isScript = path.endsWith(".script.yaml");
if (!isScript) {
return content;
}
let isTs = true;
try {
await Deno.stat(path.replace(".script.yaml", ".ts"));
} catch {
isTs = false;
}
if (!isTs) {
return content;
}
log.info(`isScript: ${isScript}, isTs: ${isTs}; ${path}`);
if (isTs) {
const c = findCodebase(path, codebases);
if (c) {
const parsed: any = yamlParse(content);
if (parsed && typeof parsed == "object") {
parsed["codebase"] = c.digest;
return yamlStringify(parsed, yamlOptions);
} else {
throw Error(
`Expected local yaml ${path} to be an object, found: ${content} instead`
);
}
}
}
return content;
}
export async function FSFSElement(
p: string,
codebases: SyncCodebase[]
): Promise<DynFSElement> {
function _internal_element(
localP: string,
isDir: boolean,
codebases: SyncCodebase[]
): DynFSElement {
return {
isDirectory: isDir,
path: localP.substring(p.length + 1),
@@ -63,7 +141,11 @@ export async function FSFSElement(p: string): Promise<DynFSElement> {
if (!isDir) return [];
try {
for await (const e of Deno.readDir(localP)) {
yield _internal_element(path.join(localP, e.name), e.isDirectory);
yield _internal_element(
path.join(localP, e.name),
e.isDirectory,
codebases
);
}
} catch (e) {
log.warning(`Error reading dir: ${localP}, ${e}`);
@@ -74,11 +156,12 @@ export async function FSFSElement(p: string): Promise<DynFSElement> {
// },
async getContentText(): Promise<string> {
const content = await Deno.readTextFile(localP);
return content;
return await addCodebaseDigestIfRelevant(localP, content, codebases);
},
};
}
return _internal_element(p, (await Deno.stat(p)).isDirectory);
return _internal_element(p, (await Deno.stat(p)).isDirectory, codebases);
}
function prioritizeName(name: string): string {
@@ -727,6 +810,8 @@ async function pull(opts: GlobalOptions & SyncOptions) {
const workspace = await resolveWorkspace(opts);
await requireLogin(opts);
const codebases = await listSyncCodebases(opts);
log.info(
colors.gray(
"Computing the files to update locally to match remote (taking wmill.yaml into account)"
@@ -748,8 +833,8 @@ async function pull(opts: GlobalOptions & SyncOptions) {
!opts.json
);
const local = !opts.stateful
? await FSFSElement(Deno.cwd())
: await FSFSElement(path.join(Deno.cwd(), ".wmill"));
? await FSFSElement(Deno.cwd(), codebases)
: await FSFSElement(path.join(Deno.cwd(), ".wmill"), []);
const changes = await compareDynFSElement(
remote,
local,
@@ -924,6 +1009,7 @@ function removeSuffix(str: string, suffix: string) {
async function push(opts: GlobalOptions & SyncOptions) {
opts = await mergeConfigWithConfigFile(opts);
const codebases = await listSyncCodebases(opts);
if (opts.raw) {
log.info("--raw is now the default, you can remove it as a flag");
}
@@ -962,7 +1048,7 @@ async function push(opts: GlobalOptions & SyncOptions) {
!opts.json
);
const local = await FSFSElement(path.join(Deno.cwd(), ""));
const local = await FSFSElement(path.join(Deno.cwd(), ""), codebases);
const changes = await compareDynFSElement(
local,
remote,
@@ -991,10 +1077,11 @@ async function push(opts: GlobalOptions & SyncOptions) {
) {
return;
}
log.info(colors.gray(`Applying changes to files ...`));
const alreadySynced: string[] = [];
const globalDeps = await findGlobalDeps();
const globalDeps = await findGlobalDeps(codebases);
for await (const change of changes) {
const stateTarget = path.join(Deno.cwd(), ".wmill", change.path);
@@ -1012,7 +1099,9 @@ async function push(opts: GlobalOptions & SyncOptions) {
workspace,
alreadySynced,
opts.message,
globalDeps
globalDeps,
codebases,
opts
)
) {
if (opts.stateful && stateExists) {
@@ -1026,7 +1115,8 @@ async function push(opts: GlobalOptions & SyncOptions) {
alreadySynced,
opts.message,
opts,
globalDeps
globalDeps,
codebases
)
) {
if (opts.stateful && stateExists) {
@@ -1067,7 +1157,8 @@ async function push(opts: GlobalOptions & SyncOptions) {
alreadySynced,
opts.message,
opts,
globalDeps
globalDeps,
codebases
)
) {
continue;
cli/utils.ts
+26 -1
View File
@@ -2,7 +2,7 @@
// eslint-disable-next-line @typescript-eslint/ban-ts-comment
// @ts-nocheck This file is copied from a JS project, so it's not type-safe.
import { log } from "./deps.ts";
import { log, encodeHex } from "./deps.ts";
export function deepEqual<T>(a: T, b: T): boolean {
if (a === b) return true;
@@ -92,3 +92,28 @@ export function getHeaders(): Record<string, string> | undefined {
return undefined;
}
}
export async function digestDir(path: string) {
const hashes: string = [];
for await (const e of Deno.readDir(path)) {
const npath = path + "/" + e.name;
if (e.isFile) {
hashes.push(await generateHashFromBuffer(await Deno.readFile(npath)));
} else if (e.isDirectory && !e.isSymlink) {
hashes.push(await digestDir(npath));
}
}
return await generateHash(hashes.join(""));
}
export async function generateHash(content: string): Promise<string> {
const messageBuffer = new TextEncoder().encode(content);
return await generateHashFromBuffer(messageBuffer);
}
export async function generateHashFromBuffer(
content: BufferSource
): Promise<string> {
const hashBuffer = await crypto.subtle.digest("SHA-256", content);
return encodeHex(hashBuffer);
}
docker-compose.yml
+6 -1
View File
@@ -37,6 +37,8 @@ services:
depends_on:
db:
condition: service_healthy
volumes:
- worker_logs:/tmp/windmill/logs
windmill_worker:
image: ${WM_IMAGE}
@@ -60,6 +62,7 @@ services:
# mount the docker socket to allow to run docker containers from within the workers
- /var/run/docker.sock:/var/run/docker.sock
- worker_dependency_cache:/tmp/windmill/cache
- worker_logs:/tmp/windmill/logs
## This worker is specialized for "native" jobs. Native jobs run in-process and thus are much more lightweight than other jobs
windmill_worker_native:
@@ -80,7 +83,8 @@ services:
depends_on:
db:
condition: service_healthy
volumes:
- worker_logs:/tmp/windmill/logs
## This worker is specialized for reports or scraping jobs. It is assigned the "reports" worker group which has an init script that installs chromium and can be targeted by using the "chromium" worker tag.
# windmill_worker_reports:
# image: ${WM_IMAGE}
@@ -142,4 +146,5 @@ services:
volumes:
db_data: null
worker_dependency_cache: null
worker_logs: null
lsp_cache: null
frontend/package-lock.json
Generated
+2 -2
View File
@@ -1,12 +1,12 @@
{
"name": "windmill-components",
"version": "1.323.6",
"version": "1.327.0",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "windmill-components",
"version": "1.323.6",
"version": "1.327.0",
"license": "AGPL-3.0",
"dependencies": {
"@aws-crypto/sha256-js": "^4.0.0",
frontend/package.json
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "windmill-components",
"version": "1.323.6",
"version": "1.327.0",
"scripts": {
"dev": "vite dev",
"build": "vite build",
frontend/src/lib/assets/app.css
+1
View File
@@ -157,4 +157,5 @@
.app-editor-input {
@apply rounded-component border border-gray-300 dark:border-gray-500 focus:border-gray-300 focus:dark:border-gray-500 focus:!ring-1 focus:!ring-blue-300;
@apply placeholder:text-gray-400 dark:placeholder:text-gray-600;
}
frontend/src/lib/autosize.ts
+1 -1
View File
@@ -44,7 +44,7 @@ export const action = (node) => {
const setHeight = () => {
node.style.height = '0px'
node.style.height = Math.max(node.scrollHeight ?? 0, 40) + 2 + 'px'
node.style.height = Math.max(node.scrollHeight ?? 0, 30) + 2 + 'px'
}
const addStyles = () => {
frontend/src/lib/components/AppConnect.svelte
+10 -2
View File
@@ -447,9 +447,17 @@
<h2 class="mt-8 mb-4">Others</h2>
{#if connectsManual && connectsManual?.length < 10}
<div class="text-tertiary p-2">
<div class="text-secondary p-2">
Resource Types have not been synced with the hub. Go to the admins workspace to sync them
(and add a schedule to do daily).
(and add a schedule to do daily):
<p class="mt-4"
>1. Go to the "admins" workspaces:
<img src="/sync_resource_types.png" alt="sync resource types" class="mt-2" />
</p>
<p class="mt-4">
2: Run the synchronization script:
<img src="/sync_resource_types2.png" alt="sync resource types" class="mt-2" />
</p>
</div>
{/if}
frontend/src/lib/components/ArgInput.svelte
+1
View File
@@ -550,6 +550,7 @@
{#if properties && Object.keys(properties).length > 0}
<div class="p-4 pl-8 border rounded w-full">
<SchemaForm
{onlyMaskPassword}
{disablePortal}
{disabled}
schema={{ properties, $schema: '', required: nestedRequired ?? [], type: 'object' }}
frontend/src/lib/components/DateTimeInput.svelte
+10
View File
@@ -2,6 +2,8 @@
import { createEventDispatcher } from 'svelte'
import { Button } from './common'
import { Clock } from 'lucide-svelte'
// import ToggleButtonGroup from './common/toggleButton-v2/ToggleButtonGroup.svelte'
// import ToggleButton from './common/toggleButton-v2/ToggleButton.svelte'
export let value: string | undefined = undefined
@@ -13,6 +15,8 @@
let date: string | undefined = undefined
let time: string | undefined = undefined
// let format: 'local' | 'utc' = 'local'
function parseValue(value: string | undefined = undefined) {
let dateFromValue: Date | undefined = value ? new Date(value) : undefined
date = isValidDate(dateFromValue)
@@ -115,4 +119,10 @@
>
Now
</Button>
<!-- <div>
<ToggleButtonGroup bind:selected={format}>
<ToggleButton light small value={'local'} label="local" />
<ToggleButton light small value={'utc'} label="utc" />
</ToggleButtonGroup>
</div> -->
</div>
frontend/src/lib/components/Dev.svelte
+86 -16
View File
@@ -159,6 +159,8 @@
path: string
language: Preview['language']
lock?: string
isCodebase?: boolean
tag?: string
}
let currentScript: LastEditScript | undefined = undefined
@@ -177,13 +179,28 @@
let lockChanges = false
let timeout: NodeJS.Timeout | undefined = undefined
let loadingCodebaseButton = false
let lastBundleCommandId = ''
const el = (event) => {
// sendUserToast(`Received message from parent ${event.data.type}`, true)
if (event.data.type == 'runTest') {
runTest()
event.preventDefault()
} else if (event.data.type == 'replaceScript') {
mode = 'script'
replaceScript(event.data)
} else if (event.data.type == 'testBundle') {
if (event.data.id == lastBundleCommandId) {
testBundle(event.data.file)
} else {
sendUserToast(`Bundle received ${lastBundleCommandId} was obsolete, ignoring`, true)
}
} else if (event.data.type == 'testBundleError') {
sendUserToast(
typeof event.data.error == 'object' ? JSON.stringify(event.data.error) : event.data.error,
true
)
} else if (event.data.type == 'replaceFlow') {
mode = 'flow'
lockChanges = true
@@ -223,6 +240,47 @@
window.parent?.postMessage({ type: 'refresh' }, '*')
})
async function testBundle(file: string) {
testJobLoader?.abstractRun(async () => {
try {
const form = new FormData()
form.append(
'preview',
JSON.stringify({
content: currentScript?.content,
kind: 'bundle',
path: currentScript?.path,
args,
language: currentScript?.language,
tag: currentScript?.tag
})
)
form.append('file', file)
const url = '/api/w/' + workspace + '/jobs/run/preview_bundle'
const req = await fetch(url, {
method: 'POST',
body: form,
headers: {
Authorization: 'Bearer ' + token
}
})
if (req.status != 201) {
throw Error(
`Script snapshot creation was not successful: ${req.status} - ${
req.statusText
} - ${await req.text()}`
)
}
return await req.text()
} catch (e) {
sendUserToast(`Failed to send bundle ${e}`, true)
throw Error(e)
}
})
loadingCodebaseButton = false
}
onDestroy(() => {
window.removeEventListener('message', el)
if (socket && socket.readyState === WebSocket.OPEN) {
@@ -261,15 +319,21 @@
if (!currentScript) {
return
}
//@ts-ignore
testJobLoader.runPreview(
currentScript.path,
currentScript.content,
currentScript.language,
args,
undefined,
useLock ? currentScript.lock : undefined
)
if (currentScript.isCodebase) {
loadingCodebaseButton = true
lastBundleCommandId = Math.random().toString(36).substring(7)
window.parent?.postMessage({ type: 'testBundle', id: lastBundleCommandId }, '*')
} else {
//@ts-ignore
testJobLoader.runPreview(
currentScript.path,
currentScript.content,
currentScript.language,
args,
currentScript.tag,
useLock ? currentScript.lock : undefined
)
}
} else {
flowPreviewButtons?.openPreview()
}
@@ -288,10 +352,11 @@
})
}
function onKeyDown(event: KeyboardEvent) {
if ((event.ctrlKey || event.metaKey) && event.key == 'Enter') {
event.preventDefault()
runTest()
async function onKeyDown(event: KeyboardEvent) {
if ((event.ctrlKey || event.metaKey) && event.code === 'KeyC') {
document.execCommand('copy')
} else if ((event.ctrlKey || event.metaKey) && event.code === 'KeyX') {
document.execCommand('cut')
}
}
@@ -445,7 +510,9 @@
</div>
<div class="text-center w-full text-lg truncate py-1 text-primary">
{currentScript?.path ?? 'Not editing a script'}
{currentScript?.language ?? ''}
<span class="text-2xs text-secondary">{currentScript?.language ?? ''}</span><span
class="text-2xs text-secondary">{currentScript?.isCodebase ? ' (codebase)' : ''}</span
>
</div>
<div class="absolute top-2 right-2 !text-tertiary text-xs">
@@ -482,7 +549,7 @@
<Button
disabled={currentScript === undefined}
color="dark"
on:click={() => {
on:click={(e) => {
runTest()
}}
btnClasses="w-full"
@@ -493,7 +560,9 @@
}}
shortCut={{ Icon: CornerDownLeft, hide: testIsLoading }}
>
{#if testIsLoading}
{#if loadingCodebaseButton}
Bundle is building...
{:else if testIsLoading}
Running
{:else}
Test
@@ -511,6 +580,7 @@
</Pane>
<Pane size={67}>
<LogPanel
{workspace}
lang={currentScript?.language}
previewJob={testJob}
{pastPreviews}
frontend/src/lib/components/DisplayResult.svelte
+18 -12
View File
@@ -11,7 +11,8 @@
PanelRightOpen,
Table2,
Braces,
Highlighter
Highlighter,
InfoIcon
} from 'lucide-svelte'
import Portal from 'svelte-portal'
import ObjectViewer from './propertyPicker/ObjectViewer.svelte'
@@ -26,7 +27,7 @@
import ToggleButtonGroup from './common/toggleButton-v2/ToggleButtonGroup.svelte'
import ToggleButton from './common/toggleButton-v2/ToggleButton.svelte'
import MapResult from './MapResult.svelte'
import Tooltip from '$lib/components/Tooltip.svelte'
import Popover from './Popover.svelte'
export let result: any
export let requireHtmlApproval = false
@@ -36,6 +37,7 @@
export let workspaceId: string | undefined = undefined
export let hideAsJson: boolean = false
export let noControls: boolean = false
export let drawerOpen = false
let resultKind:
| 'json'
@@ -272,7 +274,7 @@
: 'min-h-[200px]'}"
>
{#if result != undefined && length != undefined && largeObject != undefined}
<div class="flex justify-between items-center w-full pb-1">
<div class="flex justify-between items-center w-full">
<div class="text-tertiary text-sm flex items-center">
{#if (resultKind && typeof result == 'object' && !['json', 's3object', 's3object-list', 'table-col', 'table-row'].includes(resultKind) && !hideAsJson) || isTableDisplay}
<ToggleButtonGroup
@@ -301,21 +303,25 @@
</ToggleButtonGroup>
{/if}
</div>
<div class="text-tertiary text-xs flex gap-2 z-10 items-center">
<div class="text-secondary text-xs flex gap-2.5 z-10 items-center">
<slot name="copilot-fix" />
{#if !disableExpand && !noControls}
<Tooltip
<Popover
documentationLink="https://www.windmill.dev/docs/core_concepts/rich_display_rendering"
customSize="115%"
>
The result renderer in Windmill supports rich display rendering, allowing you to
customize the display format of your results.
</Tooltip>
<svelte:fragment slot="text">
The result renderer in Windmill supports rich display rendering, allowing you to
customize the display format of your results.
</svelte:fragment>
<div class="-mt-1">
<InfoIcon size={14} />
</div>
</Popover>
<button on:click={() => copyToClipboard(toJsonStr(result))} class="-mt-1">
<ClipboardCopy size={16} />
<ClipboardCopy size={14} />
</button>
<button on:click={jsonViewer.openDrawer} class="-mt-1">
<Expand size={16} />
<Expand size={14} />
</button>
{/if}
</div>
@@ -598,7 +604,7 @@
</div>
{#if !disableExpand && !noControls}
<Drawer bind:this={jsonViewer} size="900px">
<Drawer bind:this={jsonViewer} bind:open={drawerOpen} size="900px">
<DrawerContent title="Expanded Result" on:close={jsonViewer.closeDrawer}>
<svelte:fragment slot="actions">
<Button
frontend/src/lib/components/Editor.svelte
+14 -1
View File
@@ -27,6 +27,8 @@
import libStdContent from '$lib/es6.d.ts.txt?raw'
import denoFetchContent from '$lib/deno_fetch.d.ts.txt?raw'
import processStdContent from '$lib/process.d.ts.txt?raw'
import windmillFetchContent from '$lib/windmill_fetch.d.ts.txt?raw'
import { MonacoLanguageClient } from 'monaco-languageclient'
@@ -757,10 +759,20 @@
} else if (lang === 'typescript') {
const stdLib = { content: libStdContent, filePath: 'es6.d.ts' }
if (scriptLang == 'bun') {
languages.typescript.typescriptDefaults.setExtraLibs([stdLib])
const processLib = { content: processStdContent, filePath: 'process.d.ts' }
languages.typescript.typescriptDefaults.setExtraLibs([stdLib, processLib])
} else {
const denoFetch = { content: denoFetchContent, filePath: 'deno_fetch.d.ts' }
languages.typescript.typescriptDefaults.setExtraLibs([stdLib, denoFetch])
let localContent = windmillFetchContent
let p = '/tmp/monaco/windmill.d.ts'
let nuri = mUri.parse(p)
let localModel = meditor.getModel(nuri)
if (localModel) {
localModel.setValue(localContent)
} else {
meditor.createModel(localContent, 'typescript', nuri)
}
}
if (scriptLang == 'bun' && ata == undefined) {
const addLibraryToRuntime = async (code: string, _path: string) => {
@@ -1051,6 +1063,7 @@
noUnusedLocals: true,
strict: true,
noLib: false,
allowImportingTsExtensions: true,
moduleResolution: languages.typescript.ModuleResolutionKind.NodeJs
})
frontend/src/lib/components/EditorBar.svelte
+12 -18
View File
@@ -269,8 +269,8 @@
if (!editor) return
if (lang == 'deno') {
editor.insertAtCursor(`Deno.env.get('${name}')`)
} else if (lang === 'bun') {
editor.insertAtCursor(`Bun.env["${name}"]`)
} else if (lang === 'bun' || lang == 'nativets') {
editor.insertAtCursor(`process.env["${name}"]`)
} else if (lang == 'python3') {
if (!editor.getCode().includes('import os')) {
editor.insertAtBeginning('import os\n')
@@ -285,8 +285,6 @@
editor.insertAtCursor(`$${name}`)
} else if (lang == 'powershell') {
editor.insertAtCursor(`$Env:${name}`)
} else if (lang == 'nativets') {
editor.insertAtCursor(name)
}
sendUserToast(`${name} inserted at cursor`)
}}
@@ -333,13 +331,11 @@
`\nInvoke-RestMethod -Headers $Headers -Uri "$Env:BASE_INTERNAL_URL/api/w/$Env:WM_WORKSPACE/variables/get_value/${path}"`
)
} else if (lang == 'nativets') {
editor.insertAtCursor(
'await fetch(`${BASE_INTERNAL_URL}/api/w/${WM_WORKSPACE}/variables/get_value/' +
path +
'`, {\nheaders: { Authorization: `Bearer ${WM_TOKEN}` }'
)
editor.arrowDown()
editor.insertAtCursor('.then(res => res.json())')
const code = editor.getCode()
if (!code.includes(`import * as wmill from`)) {
editor.insertAtBeginning(`import * as wmill from "./windmill.ts"\n`)
}
editor.insertAtCursor(`(await wmill.getVariable('${path}'))`)
}
sendUserToast(`${name} inserted at cursor`)
}}
@@ -400,13 +396,11 @@
`\nInvoke-RestMethod -Headers $Headers -Uri "$Env:BASE_INTERNAL_URL/api/w/$Env:WM_WORKSPACE/resources/get_value_interpolated/${path}"`
)
} else if (lang == 'nativets') {
editor.insertAtCursor(
'await fetch(`${BASE_INTERNAL_URL}/api/w/${WM_WORKSPACE}/resources/get_value_interpolated/' +
path +
'`, {\nheaders: { Authorization: `Bearer ${WM_TOKEN}` }'
)
editor.arrowDown()
editor.insertAtCursor('.then(res => res.json())')
const code = editor.getCode()
if (!code.includes(`import * as wmill from`)) {
editor.insertAtBeginning(`import * as wmill from "./windmill.ts"\n`)
}
editor.insertAtCursor(`(await wmill.getResource('${path}'))`)
}
sendUserToast(`${path} inserted at cursor`)
}}
frontend/src/lib/components/FlowGraphViewerStep.svelte
+19 -1
View File
@@ -10,11 +10,12 @@
import FlowPathViewer from './flows/content/FlowPathViewer.svelte'
import SchemaViewer from './SchemaViewer.svelte'
import { scriptPathToHref } from '$lib/scripts'
import { cleanExpr } from '$lib/utils'
import { cleanExpr, copyToClipboard } from '$lib/utils'
import { hubBaseUrlStore } from '$lib/stores'
import { twMerge } from 'tailwind-merge'
import FlowModuleScript from './flows/content/FlowModuleScript.svelte'
import { Copy } from 'lucide-svelte'
export let flow: {
summary: string
@@ -212,6 +213,23 @@
{/each}
</div>
{:else if stepDetail.value.type == 'flow'}
<div class="text-sm mb-1 flex justify-end flex-row">
<Button
size="xs2"
startIcon={{ icon: Copy }}
color="light"
on:click={() => {
if (typeof stepDetail !== 'string') {
const val = stepDetail?.value
if (val?.type == 'flow') {
copyToClipboard(val.path)
}
}
}}
>
Copy path
</Button>
</div>
<FlowPathViewer noSide path={stepDetail.value.path} />
{/if}
{:else}
frontend/src/lib/components/FlowMetadata.svelte
+4 -6
View File
@@ -11,11 +11,9 @@
export let job: Job
const SMALL_ICON_SIZE = 14
let scheduleEditor: ScheduleEditor
export let scheduleEditor: ScheduleEditor
</script>
<ScheduleEditor bind:this={scheduleEditor} />
<div
class="rounded-md p-3 bg-surface-secondary shadow-sm sm:text-sm md:text-base"
style="min-height: 150px;"
@@ -88,19 +86,19 @@
{@const stem = `/${job?.job_kind}s`}
{@const isScript = job?.job_kind === 'script'}
{@const viewHref = `${stem}/get/${isScript ? job?.script_hash : job?.script_path}`}
<div class="flex flex-row gap-2 items-center">
<div class="flex flex-row gap-2 items-center ml-0.5">
{#if isScript}
<Code2 size={SMALL_ICON_SIZE} class="text-secondary" />
{:else}
<BarsStaggered size={SMALL_ICON_SIZE} class="text-secondary" />
{/if}
<span>
<span class="truncate text-sm">
<a href={viewHref}>{isScript ? job?.script_hash : job?.script_path}</a>
</span>
</div>
{/if}
<div class="flex flex-row gap-2 items-center">
<div class="flex flex-row gap-2 items-center text-sm">
<User size={SMALL_ICON_SIZE} class="text-secondary" />
<span>
frontend/src/lib/components/JobStatus.svelte
+1 -1
View File
@@ -41,7 +41,7 @@
</div>
{:else if job && 'running' in job}
<div class="flex flex-row gap-1 items-center">
<Badge icon={{ icon: Clock, position: 'left' }}>Queued</Badge>
<Badge color="orange" icon={{ icon: Clock, position: 'left' }}>Queued</Badge>
<NoWorkerWithTagWarning tag={job.tag} />
</div>
{:else}
frontend/src/lib/components/LightweightArgInput.svelte
+302 -281
View File
@@ -49,6 +49,7 @@
export let customErrorMessage: string | undefined = undefined
export let hideResourceInput: boolean = false
export let resourceInputUnsupported: boolean = false
export let render = true
const dispatch = createEventDispatcher()
@@ -60,7 +61,7 @@
$: inputCat = computeInputCat(type, format, itemsType?.type, enum_, contentEncoding)
$: changeDefaultValue(inputCat, defaultValue)
$: render && changeDefaultValue(inputCat, defaultValue)
$: rawValue && evalRawValueToValue()
@@ -111,6 +112,10 @@
}
function validateInput(pattern: string | undefined, v: any, required: boolean): void {
if (!render) {
error = ''
value = true
}
if (required && (v == undefined || v == null || v === '')) {
error = 'Required'
valid && (valid = false)
@@ -158,304 +163,320 @@
evalValueToRaw()
}
}
$: render == false && onRenderFalse()
function onRenderFalse() {
value = undefined
valid = true
error = ''
}
</script>
{#if !(hideResourceInput && inputCat === 'resource-object')}
<div class="flex flex-col w-full min-w-[250px]">
<div>
{#if displayHeader}
<FieldHeader
prettify
{label}
{required}
{type}
{contentEncoding}
{format}
{displayType}
labelClass={css?.label?.class}
/>
{/if}
{#if description}
<div class={twMerge('text-xs italic pb-1', css?.description?.class)}>
<pre class="font-main whitespace-normal">{description}</pre>
</div>
{/if}
<div class="flex space-x-1">
{#if inputCat == 'number'}
{#if extra['min'] != undefined && extra['max'] != undefined}
<div class="flex w-full gap-1">
<span>{extra['min']}</span>
<div class="grow">
<Range bind:value min={extra['min']} max={extra['max']} />
</div>
<span>{extra['max']}</span>
<span class="mx-2"><Badge large color="blue">{value}</Badge></span>
</div>
{:else if extra?.currency}
<CurrencyInput
inputClasses={{
formatted: 'px-2 w-full py-1.5 text-black dark:text-white',
wrapper: 'w-full windmillapp',
formattedZero: 'text-black dark:text-white'
}}
noColor
bind:value
currency={extra?.currency}
locale={extra?.currencyLocale ?? 'en-US'}
/>
{:else}
<input
on:focus={(e) => {
dispatch('focus')
}}
type="number"
class={twMerge(
valid && error == ''
? ''
: 'border !border-red-700 !border-opacity-70 focus:!border-red-700 focus:!border-opacity-30'
)}
placeholder={defaultValue ?? ''}
bind:value
min={extra['min']}
max={extra['max']}
/>
{/if}
{:else if inputCat == 'boolean'}
<Toggle
on:pointerdown={(e) => {
e?.stopPropagation()
}}
class={valid && error == ''
? ''
: 'border !border-red-700 !border-opacity-70 focus:!border-red-700 focus:!border-opacity-30'}
bind:checked={value}
{#if render}
{#if !(hideResourceInput && inputCat === 'resource-object')}
<div class="flex flex-col w-full min-w-[250px]">
<div>
{#if displayHeader}
<FieldHeader
prettify
{label}
{required}
{type}
{contentEncoding}
{format}
{displayType}
labelClass={css?.label?.class}
/>
{#if type == 'boolean' && value == undefined}
<span>&nbsp; Not set</span>
{/if}
{:else if inputCat == 'list'}
<div class="w-full">
{#if Array.isArray(itemsType?.multiselect) && Array.isArray(value)}
<div class="items-start">
<Multiselect
ulOptionsClass={'!bg-surface-secondary'}
bind:selected={value}
options={itemsType?.multiselect ?? []}
selectedOptionsDraggable={true}
/>
{/if}
{#if description}
<div class={twMerge('text-xs italic pb-1', css?.description?.class)}>
<pre class="font-main whitespace-normal">{description}</pre>
</div>
{/if}
<div class="flex space-x-1">
{#if inputCat == 'number'}
{#if extra['min'] != undefined && extra['max'] != undefined}
<div class="flex w-full gap-1">
<span>{extra['min']}</span>
<div class="grow">
<Range bind:value min={extra['min']} max={extra['max']} />
</div>
<span>{extra['max']}</span>
<span class="mx-2"><Badge large color="blue">{value}</Badge></span>
</div>
{:else if Array.isArray(itemsType?.enum) && Array.isArray(value)}
<div class="items-start">
<Multiselect
ulOptionsClass={'!bg-surface-secondary'}
bind:selected={value}
options={itemsType?.enum ?? []}
selectedOptionsDraggable={true}
/>
</div>
{:else if Array.isArray(enum_) && Array.isArray(value)}
<div class="items-start">
<Multiselect
ulOptionsClass={'!bg-surface-secondary'}
bind:selected={value}
options={enum_ ?? []}
selectedOptionsDraggable={true}
{:else if extra?.currency}
<CurrencyInput
inputClasses={{
formatted: 'px-2 w-full py-1.5 text-black dark:text-white',
wrapper: 'w-full windmillapp',
formattedZero: 'text-black dark:text-white'
}}
noColor
bind:value
currency={extra?.currency}
locale={extra?.currencyLocale ?? 'en-US'}
/>
{:else}
<input
on:focus={(e) => {
dispatch('focus')
}}
type="number"
class={twMerge(
valid && error == ''
? ''
: 'border !border-red-700 !border-opacity-70 focus:!border-red-700 focus:!border-opacity-30'
)}
placeholder={defaultValue ?? ''}
bind:value
min={extra['min']}
max={extra['max']}
/>
{/if}
{:else if inputCat == 'boolean'}
<Toggle
on:pointerdown={(e) => {
e?.stopPropagation()
}}
class={valid && error == ''
? ''
: 'border !border-red-700 !border-opacity-70 focus:!border-red-700 focus:!border-opacity-30'}
bind:checked={value}
/>
{#if type == 'boolean' && value == undefined}
<span>&nbsp; Not set</span>
{/if}
{:else if inputCat == 'list'}
<div class="w-full">
{#if Array.isArray(itemsType?.multiselect) && Array.isArray(value)}
<div class="items-start">
<Multiselect
ulOptionsClass={'!bg-surface-secondary'}
bind:selected={value}
options={itemsType?.multiselect ?? []}
selectedOptionsDraggable={true}
/>
</div>
{:else if Array.isArray(itemsType?.enum) && Array.isArray(value)}
<div class="items-start">
<Multiselect
ulOptionsClass={'!bg-surface-secondary'}
bind:selected={value}
options={itemsType?.enum ?? []}
selectedOptionsDraggable={true}
/>
</div>
{:else if Array.isArray(enum_) && Array.isArray(value)}
<div class="items-start">
<Multiselect
ulOptionsClass={'!bg-surface-secondary'}
bind:selected={value}
options={enum_ ?? []}
selectedOptionsDraggable={true}
/>
</div>
{:else}
<div class="w-full">
{#if Array.isArray(value)}
{#each value ?? [] as v, i}
<div class="flex flex-row max-w-md mt-1 w-full">
{#if itemsType?.type == 'number'}
<input type="number" bind:value={v} />
{:else if itemsType?.type == 'string' && itemsType?.contentEncoding == 'base64'}
<input
type="file"
class="my-6"
on:change={(x) => fileChanged(x, (val) => (value[i] = val))}
multiple={false}
/>
{:else if Array.isArray(itemsType?.enum)}
<select
on:focus={(e) => {
dispatch('focus')
}}
class="px-6"
bind:value={v}
>
{#each itemsType?.enum ?? [] as e}
<option>{e}</option>
{/each}
</select>
{:else}
<input type="text" bind:value={v} />
{/if}
<button
transition:fade|local={{ duration: 100 }}
class="rounded-full p-1 bg-surface-secondary duration-200 hover:bg-surface-hover ml-2"
aria-label="Clear"
on:click={() => {
value = value.filter((el) => el != v)
if (value.length == 0) {
value = undefined
}
}}
>
<X size={14} />
</button>
</div>
{/each}
{:else if value != undefined}
List is not an array
{/if}
</div>
<div class="flex my-2">
<Button
variant="border"
color="light"
size="sm"
btnClasses="mt-1"
on:click={() => {
if (value == undefined || !Array.isArray(value)) {
value = []
}
value = value.concat('')
}}
startIcon={{ icon: Plus }}
>
Add
</Button>
</div>
<span class="ml-2">
{(value ?? []).length} item{(value ?? []).length != 1 ? 's' : ''}
</span>
{/if}
</div>
{:else if inputCat == 'resource-object'}
<LightweightObjectResourceInput
{format}
bind:value
unsupported={resourceInputUnsupported}
/>
{:else if inputCat == 'object'}
{#if properties && Object.keys(properties).length > 0}
<div class="p-4 pl-8 border rounded w-full">
<LightweightSchemaForm
schema={{
properties,
$schema: '',
required: nestedRequired ?? [],
type: 'object'
}}
bind:args={value}
/>
</div>
{:else}
<div class="w-full">
{#if Array.isArray(value)}
{#each value ?? [] as v, i}
<div class="flex flex-row max-w-md mt-1 w-full">
{#if itemsType?.type == 'number'}
<input type="number" bind:value={v} />
{:else if itemsType?.type == 'string' && itemsType?.contentEncoding == 'base64'}
<input
type="file"
class="my-6"
on:change={(x) => fileChanged(x, (val) => (value[i] = val))}
multiple={false}
/>
{:else if Array.isArray(itemsType?.enum)}
<select
on:focus={(e) => {
dispatch('focus')
}}
class="px-6"
bind:value={v}
>
{#each itemsType?.enum ?? [] as e}
<option>{e}</option>
{/each}
</select>
{:else}
<input type="text" bind:value={v} />
{/if}
<button
transition:fade|local={{ duration: 100 }}
class="rounded-full p-1 bg-surface-secondary duration-200 hover:bg-surface-hover ml-2"
aria-label="Clear"
on:click={() => {
value = value.filter((el) => el != v)
if (value.length == 0) {
value = undefined
}
}}
>
<X size={14} />
</button>
</div>
{/each}
{:else if value != undefined}
List is not an array
{/if}
</div>
<div class="flex my-2">
<Button
variant="border"
color="light"
size="sm"
btnClasses="mt-1"
on:click={() => {
if (value == undefined || !Array.isArray(value)) {
value = []
}
value = value.concat('')
}}
startIcon={{ icon: Plus }}
>
Add
</Button>
</div>
<span class="ml-2">
{(value ?? []).length} item{(value ?? []).length != 1 ? 's' : ''}
</span>
{/if}
</div>
{:else if inputCat == 'resource-object'}
<LightweightObjectResourceInput
{format}
bind:value
unsupported={resourceInputUnsupported}
/>
{:else if inputCat == 'object'}
{#if properties && Object.keys(properties).length > 0}
<div class="p-4 pl-8 border rounded w-full">
<LightweightSchemaForm
schema={{ properties, $schema: '', required: nestedRequired ?? [], type: 'object' }}
bind:args={value}
<textarea
bind:this={el}
on:focus={(e) => {
dispatch('focus')
}}
use:autosize
style="min-height: 5px;"
class="col-span-10 {valid && error == ''
? ''
: 'border !border-red-700 !border-opacity-70 focus:!border-red-700 focus:!border-opacity-30'}"
placeholder={defaultValue ? JSON.stringify(defaultValue, null, 4) : ''}
bind:value={rawValue}
/>
</div>
{:else}
<textarea
bind:this={el}
{/if}
{:else if inputCat == 'enum'}
<select
on:focus={(e) => {
dispatch('focus')
}}
use:autosize
style="min-height: 5px;"
class="col-span-10 {valid && error == ''
? ''
: 'border !border-red-700 !border-opacity-70 focus:!border-red-700 focus:!border-opacity-30'}"
placeholder={defaultValue ? JSON.stringify(defaultValue, null, 4) : ''}
bind:value={rawValue}
/>
{/if}
{:else if inputCat == 'enum'}
<select
on:focus={(e) => {
dispatch('focus')
}}
class="px-6"
bind:value
>
{#each enum_ ?? [] as e}
<option>{e}</option>
{/each}
</select>
{:else if inputCat == 'date'}
<DateTimeInput bind:value />
{:else if inputCat == 'base64'}
<div class="flex flex-col my-6 w-full">
<input
type="file"
on:change={(x) => fileChanged(x, (val) => (value = val))}
multiple={false}
/>
{#if value?.length}
<div class="text-2xs text-tertiary mt-1">File length: {value.length} base64 chars</div
>
{/if}
</div>
{:else if inputCat == 'resource-string'}
<div class="flex flex-row gap-x-1 w-full">
<LightweightResourcePicker
class="px-6"
bind:value
resourceType={format.split('-').length > 1
? format.substring('resource-'.length)
: undefined}
/>
</div>
{:else if inputCat == 'email'}
<input
on:focus
type="email"
class={valid
? ''
: 'border border-red-700 border-opacity-30 focus:border-red-700 focus:border-opacity-3'}
placeholder={defaultValue ?? ''}
bind:value
/>
{:else if inputCat == 'currency'}
<input
type="number"
class={valid
? ''
: 'border border-red-700 border-opacity-30 focus:border-red-700 focus:border-opacity-3'}
placeholder={defaultValue ?? ''}
bind:value
/>
{:else if inputCat == 'string'}
<div class="flex flex-col w-full">
<div class="flex flex-row w-full items-center justify-between">
{#if extra?.['password'] == true}
<Password bind:password={value} />
{:else}
<textarea
rows="1"
bind:this={el}
on:focus={(e) => {
dispatch('focus')
}}
use:autosize
class="col-span-10 {valid && error == ''
? ''
: 'border !border-red-700 !border-opacity-70 focus:!border-red-700 focus:!border-opacity-30'}"
placeholder={defaultValue ?? ''}
bind:value
on:pointerdown|stopPropagation={(e) => {
dispatch('inputClicked', e)
}}
/>
>
{#each enum_ ?? [] as e}
<option>{e}</option>
{/each}
</select>
{:else if inputCat == 'date'}
<DateTimeInput bind:value />
{:else if inputCat == 'base64'}
<div class="flex flex-col my-6 w-full">
<input
type="file"
on:change={(x) => fileChanged(x, (val) => (value = val))}
multiple={false}
/>
{#if value?.length}
<div class="text-2xs text-tertiary mt-1"
>File length: {value.length} base64 chars</div
>
{/if}
</div>
{:else if inputCat == 'resource-string'}
<div class="flex flex-row gap-x-1 w-full">
<LightweightResourcePicker
bind:value
resourceType={format.split('-').length > 1
? format.substring('resource-'.length)
: undefined}
/>
</div>
{:else if inputCat == 'email'}
<input
on:focus
type="email"
class={valid
? ''
: 'border border-red-700 border-opacity-30 focus:border-red-700 focus:border-opacity-3'}
placeholder={defaultValue ?? ''}
bind:value
/>
{:else if inputCat == 'currency'}
<input
type="number"
class={valid
? ''
: 'border border-red-700 border-opacity-30 focus:border-red-700 focus:border-opacity-3'}
placeholder={defaultValue ?? ''}
bind:value
/>
{:else if inputCat == 'string'}
<div class="flex flex-col w-full">
<div class="flex flex-row w-full items-center justify-between">
{#if extra?.['password'] == true}
<Password bind:password={value} />
{:else}
<textarea
rows="1"
bind:this={el}
on:focus={(e) => {
dispatch('focus')
}}
use:autosize
class="col-span-10 {valid && error == ''
? ''
: 'border !border-red-700 !border-opacity-70 focus:!border-red-700 focus:!border-opacity-30'}"
placeholder={defaultValue ?? ''}
bind:value
on:pointerdown|stopPropagation={(e) => {
dispatch('inputClicked', e)
}}
/>
{/if}
</div>
</div>
{/if}
<slot name="actions" />
</div>
{#if error && error != ''}
<div class="text-right text-xs text-red-600 dark:text-red-400">
{#if error === ''}
&nbsp;
{:else}
{error}
{/if}
</div>
{/if}
<slot name="actions" />
</div>
{#if error && error != ''}
<div class="text-right text-xs text-red-600 dark:text-red-400">
{#if error === ''}
&nbsp;
{:else}
{error}
{/if}
</div>
{/if}
</div>
</div>
{/if}
{/if}
<style>
frontend/src/lib/components/LightweightSchemaForm.svelte
+25 -26
View File
@@ -67,32 +67,31 @@
<div class={twMerge('w-full flex flex-col px-0.5 pb-2', largeGap ? 'gap-8' : 'gap-2')}>
{#each Object.keys(schema.properties ?? {}) as argName (argName)}
{#if typeof args == 'object' && schema?.properties[argName] && args}
{#if computeShow(argName, schema?.properties[argName].showExpr, args)}
<LightweightArgInput
label={argName}
description={schema.properties[argName].description}
bind:value={args[argName]}
bind:valid={inputCheck[argName]}
bind:error={errors[argName]}
type={schema.properties[argName].type}
required={schema.required?.includes(argName) ?? false}
pattern={schema.properties[argName].pattern}
defaultValue={defaultValues?.[argName] ?? schema.properties[argName].default}
enum_={dynamicEnums?.[argName] ?? schema.properties[argName].enum}
format={schema.properties[argName].format}
contentEncoding={schema.properties[argName].contentEncoding}
customErrorMessage={schema.properties[argName].customErrorMessage}
properties={schema.properties[argName].properties}
nestedRequired={schema.properties[argName].required}
itemsType={schema.properties[argName].items}
extra={schema.properties[argName]}
on:inputClicked
{displayType}
{css}
{hideResourceInput}
{resourceInputUnsupported}
/>
{/if}
<LightweightArgInput
render={computeShow(argName, schema?.properties[argName].showExpr, args)}
label={argName}
description={schema.properties[argName].description}
bind:value={args[argName]}
bind:valid={inputCheck[argName]}
bind:error={errors[argName]}
type={schema.properties[argName].type}
required={schema.required?.includes(argName) ?? false}
pattern={schema.properties[argName].pattern}
defaultValue={defaultValues?.[argName] ?? schema.properties[argName].default}
enum_={dynamicEnums?.[argName] ?? schema.properties[argName].enum}
format={schema.properties[argName].format}
contentEncoding={schema.properties[argName].contentEncoding}
customErrorMessage={schema.properties[argName].customErrorMessage}
properties={schema.properties[argName].properties}
nestedRequired={schema.properties[argName].required}
itemsType={schema.properties[argName].items}
extra={schema.properties[argName]}
on:inputClicked
{displayType}
{css}
{hideResourceInput}
{resourceInputUnsupported}
/>
{/if}
{/each}
</div>
frontend/src/lib/components/LogViewer.svelte
+41 -7
View File
@@ -1,5 +1,9 @@
<script lang="ts" context="module">
const s3LogPrefix = '[windmill] Previous logs have been saved to object storage at logs/'
const s3LogPrefixes = [
'[windmill] Previous logs have been saved to object storage at logs/',
'[windmill] Previous logs have been saved to disk at logs/',
'[windmill] No object storage set in instance settings. Previous logs have been saved to disk at '
]
</script>
<script lang="ts">
@@ -10,6 +14,7 @@
import AnsiUp from 'ansi_up'
import NoWorkerWithTagWarning from './runs/NoWorkerWithTagWarning.svelte'
import { JobService } from '$lib/gen'
import Tooltip from './Tooltip.svelte'
export let content: string | undefined
export let isLoading: boolean
@@ -19,6 +24,7 @@
export let jobId: string | undefined = undefined
export let tag: string | undefined
export let small = false
export let drawerOpen = false
// @ts-ignore
const ansi_up = new AnsiUp()
@@ -40,12 +46,39 @@
LOG_LIMIT = LOG_INC
}
$: downloadStartUrl = truncatedContent.startsWith(s3LogPrefix)
? truncatedContent.substring(s3LogPrefix.length, truncatedContent.indexOf('\n'))
: undefined
function findPrefixIndex(truncateContent: string): number | undefined {
let index = s3LogPrefixes.findIndex((x) => truncateContent.startsWith(x))
if (index == -1) {
return undefined
}
return index
}
function findStartUrl(truncateContent: string, prefixIndex: number | undefined = undefined) {
if (prefixIndex == undefined) {
return undefined
}
return prefixIndex && truncateContent
? truncateContent.substring(s3LogPrefixes[prefixIndex]?.length, truncateContent.indexOf('\n'))
: undefined
}
function tooltipText(prefixIndex: number | undefined) {
if (prefixIndex == undefined) {
return 'No path/file detected to download from'
} else if (prefixIndex == 0) {
return 'Download the previous logs from the instance configured object store'
} else if (prefixIndex == 1) {
return 'Attempt to download the logs from disk. Assume there is a shared disk between the workers and the server at /tmp/windmill/logs. Upgrade to EE to use an object store such as S3 instead of a shared volume.'
} else if (prefixIndex == 2) {
return 'Attempt to download the logs from disk. Assume there is a shared disk between the workers and the server at /tmp/windmill/logs. Since you are on EE, you can alternatively use an object store such as S3 configured in the instance settings instead of a shared volume..'
}
}
$: truncatedContent = truncateContent(content, loadedFromObjectStore, LOG_LIMIT)
$: prefixIndex = findPrefixIndex(truncatedContent)
$: downloadStartUrl = findStartUrl(truncatedContent, prefixIndex)
function truncateContent(
jobContent: string | undefined,
loadedFromObjectStore: string,
@@ -101,7 +134,7 @@
}
</script>
<Drawer bind:this={logViewer} size="900px">
<Drawer bind:this={logViewer} bind:open={drawerOpen} size="900px">
<DrawerContent title="Expanded Logs" on:close={logViewer.closeDrawer}>
<svelte:fragment slot="actions">
<Button
@@ -133,7 +166,8 @@
>{#if content}{@const len =
(content?.length ?? 0) +
(loadedFromObjectStore?.length ?? 0)}{#if downloadStartUrl}<button
on:click={getStoreLogs}>Show more...</button
on:click={getStoreLogs}
>Show more... <Tooltip>{tooltipText(prefixIndex)}</Tooltip></button
><br
/>{:else if len > LOG_LIMIT}(truncated to the last {LOG_LIMIT} characters)... <button
on:click={() => showMoreTruncate(len)}>Show more</button
@@ -199,7 +233,7 @@
>{#if content}{@const len =
(content?.length ?? 0) +
(loadedFromObjectStore?.length ?? 0)}{#if downloadStartUrl}<button on:click={getStoreLogs}
>Show more...</button
>Show more... &nbsp;<Tooltip>{tooltipText(prefixIndex)}</Tooltip></button
><br />{:else if len > LOG_LIMIT}(truncated to the last {LOG_LIMIT} characters)<br
/><button on:click={() => showMoreTruncate(len)}>Show more..</button><br />{/if}<span
>{@html html}</span
frontend/src/lib/components/OauthExtraParams.svelte
+1 -1
View File
@@ -37,7 +37,7 @@
size="sm"
endIcon={{ icon: Plus }}
on:click={() => {
extra_params_vec = extra_params_vec.concat([['key', 'value']])
extra_params_vec = (extra_params_vec ?? []).concat([['key', 'value']])
sync()
}}
>
frontend/src/lib/components/OauthScopes.svelte
+1 -1
View File
@@ -32,7 +32,7 @@
size="sm"
endIcon={{ icon: Plus }}
on:click={() => {
scopes = scopes.concat('')
scopes = (scopes ?? []).concat('')
}}
>
Add item
frontend/src/lib/components/QueueMetricsDrawer.svelte
+249
View File
@@ -0,0 +1,249 @@
<script lang="ts">
import { Drawer, DrawerContent } from './common'
import 'chartjs-adapter-date-fns'
import { Line } from 'svelte-chartjs'
import {
Chart as ChartJS,
Title,
Tooltip,
Legend,
LineElement,
CategoryScale,
LinearScale,
PointElement,
LogarithmicScale,
TimeScale,
type ChartData,
type Point
} from 'chart.js'
import { WorkerService } from '$lib/gen'
import { onDestroy } from 'svelte'
import { superadmin } from '$lib/stores'
export let drawer: Drawer
const colorTuples = [
['#7EB26D', 'rgba(126, 178, 109, 0.2)'],
['#EAB839', 'rgba(234, 184, 57, 0.2)'],
['#6ED0E0', 'rgba(110, 208, 224, 0.2)'],
['#EF843C', 'rgba(239, 132, 60, 0.2)'],
['#E24D42', 'rgba(226, 77, 66, 0.2)'],
['#1F78C1', 'rgba(31, 120, 193, 0.2)'],
['#BA43A9', 'rgba(186, 67, 169, 0.2)'],
['#705DA0', 'rgba(112, 93, 160, 0.2)'],
['#508642', 'rgba(80, 134, 66, 0.2)'],
['#CCA300', 'rgba(204, 163, 0, 0.2)'],
['#447EBC', 'rgba(68, 126, 188, 0.2)'],
['#C15C17', 'rgba(193, 92, 23, 0.2)'],
['#890F02', 'rgba(137, 15, 2, 0.2)'],
['#666666', 'rgba(102, 102, 102, 0.2)'],
['#44AA99', 'rgba(68, 170, 153, 0.2)'],
['#6D8764', 'rgba(109, 135, 100, 0.2)'],
['#555555', 'rgba(85, 85, 85, 0.2)'],
['#B3B3B3', 'rgba(179, 179, 179, 0.2)'],
['#008C9E', 'rgba(0, 140, 158, 0.2)'],
['#6BBA70', 'rgba(107, 186, 112, 0.2)']
]
function getColors(labels: string[]) {
const colors = labels.map((_, i) => colorTuples[i % colorTuples.length])
return Object.fromEntries(colors.map((c, i) => [labels[i], c]))
}
ChartJS.register(
Title,
Tooltip,
Legend,
LineElement,
LinearScale,
PointElement,
CategoryScale,
TimeScale,
LogarithmicScale
)
let countData: ChartData<'line', Point[], undefined> | undefined = undefined
let delayData: ChartData<'line', Point[], undefined> | undefined = undefined
let minDate = new Date()
let noMetrics = false
function fillData(
data: {
created_at: string
value: number
}[],
zero: number = 0
) {
let last = -1
const newElements: typeof data = []
for (const el of [
...data,
{
created_at: new Date().toISOString(),
value: zero
}
]) {
const currentTs = new Date(el.created_at).getTime()
if (last > -1 && currentTs - last > 1000 * 60 * 2) {
const numElements = Math.floor((currentTs - last) / (1000 * 30))
for (let i = 1; i < numElements; i++) {
newElements.push({
created_at: new Date(last + i * (1000 * 30)).toISOString(),
value: zero
})
}
}
last = currentTs
}
return [...data, ...newElements].sort(
(a, b) => new Date(a.created_at).getTime() - new Date(b.created_at).getTime()
)
}
async function loadMetrics() {
let metrics = await WorkerService.getQueueMetrics()
if (metrics.length == 0) {
noMetrics = true
return
}
const labels = metrics
.map((m) => m.id.slice(12))
.filter((v, i, a) => a.indexOf(v) === i)
.sort()
const labelColors = getColors(labels)
countData = {
datasets: metrics
.filter((m) => m.id.startsWith('queue_count_'))
.map((m) => {
const [color, bgColor] = labelColors[m.id.slice(12)]
return {
label: m.id.slice(12),
backgroundColor: bgColor,
borderColor: color,
data: fillData(m.values).map((v) => ({ x: v.created_at as any, y: v.value }))
}
})
}
delayData = {
datasets: metrics
.filter((m) => m.id.startsWith('queue_delay_'))
.map((m) => {
const [color, bgColor] = labelColors[m.id.slice(12)]
return {
label: m.id.slice(12),
borderColor: color,
backgroundColor: bgColor,
data: fillData(m.values, 1).map((v) => ({ x: v.created_at as any, y: v.value }))
}
})
}
minDate = new Date(Math.min(...countData.datasets.map((d) => new Date(d.data[0].x).getTime())))
}
let interval: NodeJS.Timeout | undefined = undefined
$: if ($superadmin) {
loadMetrics()
if (interval) {
clearInterval(interval)
}
interval = setInterval(loadMetrics, 35000)
} else {
if (interval) {
clearInterval(interval)
}
}
onDestroy(() => {
clearInterval(interval)
})
</script>
<Drawer bind:this={drawer}>
<DrawerContent title="Queue Metrics" on:close={drawer.closeDrawer}>
{#if noMetrics}
<p class="text-secondary">No jobs delayed by more than 3 seconds in the last 14 days</p>
{:else}
<div class="flex flex-col gap-4">
{#if countData}
<Line
data={countData}
options={{
plugins: {
title: {
display: true,
text: 'Number of delayed jobs per tag (> 3s)'
}
},
scales: {
x: {
type: 'time',
min: minDate.toISOString(),
max: new Date().toISOString()
},
y: {
title: {
display: true,
text: 'count'
}
}
}
}}
/>
{/if}
{#if delayData}
<Line
data={delayData}
options={{
plugins: {
title: {
display: true,
text: 'Queue delay per tag (> 3s)'
},
tooltip: {
callbacks: {
label: function (context) {
// @ts-ignore
if (context.raw.y === 1) {
return context.dataset.label + ': 0'
} else {
// @ts-ignore
return context.dataset.label + ': ' + context.raw.y
}
}
}
}
},
scales: {
x: {
type: 'time',
min: minDate.toISOString(),
max: new Date().toISOString()
},
y: {
type: 'logarithmic',
title: {
display: true,
text: 'delay (s)'
},
ticks: {
callback: (value, _) => (value === 1 ? '0' : value)
}
}
}
}}
/>
{/if}
</div>
{/if}
</DrawerContent>
</Drawer>
frontend/src/lib/components/SavedInputs.svelte
+48 -13
View File
@@ -8,7 +8,7 @@
type Job
} from '$lib/gen/index.js'
import { userStore, workspaceStore } from '$lib/stores.js'
import { classNames, displayDate, sendUserToast } from '$lib/utils.js'
import { classNames, displayDate, displayDateOnly, sendUserToast } from '$lib/utils.js'
import { createEventDispatcher } from 'svelte'
import { Pane, Splitpanes } from 'svelte-splitpanes'
import ObjectViewer from './propertyPicker/ObjectViewer.svelte'
@@ -17,6 +17,7 @@
import Tooltip from './Tooltip.svelte'
import TimeAgo from './TimeAgo.svelte'
import JobLoader from './runs/JobLoader.svelte'
import Skeleton from './common/skeleton/Skeleton.svelte'
export let scriptHash: string | null = null
export let scriptPath: string | null = null
@@ -32,8 +33,8 @@
isSaving?: boolean
}
let previousInputs: Input[] = []
let savedInputs: EditableInput[] = []
let previousInputs: Input[] | undefined = undefined
let savedInputs: EditableInput[] | undefined = undefined
let selectedInput: Input | null
let jobs: Job[] = []
let loading: boolean = false
@@ -92,7 +93,7 @@
is_public: false,
...requestBody
}
savedInputs = [input, ...savedInputs]
savedInputs = [input, ...(savedInputs ?? [])]
} catch (err) {
console.error(err)
sendUserToast(`Failed to save Input: ${err}`, true)
@@ -127,7 +128,7 @@
workspace: $workspaceStore!,
input: input.id
})
savedInputs = savedInputs.filter((i) => i.id !== input.id)
savedInputs = (savedInputs ?? []).filter((i) => i.id !== input.id)
if (selectedInput === input) {
selectedInput = null
}
@@ -148,6 +149,20 @@
function selectArgs(selected_args: any) {
dispatch('selected_args', selected_args)
}
async function loadLargeArgs(id: string | undefined) {
if (!id) return
largeArgs = await InputService.getArgsFromHistoryOrSavedInput({
jobOrInputId: id,
workspace: $workspaceStore!
})
}
let hasLargeArgs = false
$: hasLargeArgs =
typeof selectedInput?.args === 'string' && selectedInput?.args === 'WINDMILL_TOO_BIG'
let largeArgs: any = undefined
$: hasLargeArgs && loadLargeArgs(selectedInput?.id)
</script>
<JobLoader
@@ -192,7 +207,9 @@
</div>
<div class="w-full flex flex-col gap-2 h-full overflow-y-auto p">
{#if savedInputs.length > 0}
{#if savedInputs === undefined}
<Skeleton layout={[[8]]} />
{:else if savedInputs.length > 0}
{#each savedInputs as i}
<button
class={classNames(
@@ -290,7 +307,9 @@
<span class="text-sm font-semibold">Previous runs</span>
<div class="w-full flex flex-col gap-1 p-0 h-full overflow-y-auto">
{#if jobs.length > 0}
{#if loading}
<Skeleton layout={[[2]]} />
{:else if jobs.length > 0}
{#each jobs as i (i.id)}
<button
class={classNames(
@@ -331,7 +350,9 @@
</div>
<div class="w-full flex flex-col gap-1 p-0 h-full overflow-y-auto">
{#if previousInputs.length > 0}
{#if previousInputs === undefined}
<Skeleton layout={[[8]]} />
{:else if previousInputs.length > 0}
{#each previousInputs as i (i.id)}
<button
class={classNames(
@@ -356,11 +377,16 @@
{i.created_by}
</div>
<div
class="whitespace-nowrap col-span-3 !text-tertiary !text-2xs overflow-hidden text-ellipsis flex-shrink text-center"
class="whitespace-nowrap col-span-2 !text-tertiary !text-2xs overflow-hidden text-ellipsis flex-shrink text-center"
>
{displayDateOnly(new Date(i.created_at))}
</div>
<div
class="whitespace-nowrap col-span-2 !text-tertiary !text-2xs overflow-hidden text-ellipsis flex-shrink text-center"
>
<TimeAgo date={i.created_at ?? ''} />
</div>
<div class="col-span-2">
<div class="col-span-1">
<a
target="_blank"
href="/run/{i.id}?workspace={$workspaceStore}"
@@ -390,15 +416,24 @@
btnClasses="w-full"
size="sm"
spacingSize="xl"
on:click={() => selectArgs(selectedInput?.args)}
disabled={Object.keys(selectedInput?.args || {}).length === 0}
on:click={() => selectArgs(hasLargeArgs ? largeArgs : selectedInput?.args)}
disabled={Object.keys(selectedInput?.args || {}).length === 0 ||
(hasLargeArgs && Object.keys(largeArgs || {}).length === 0)}
>
<ArrowLeftIcon class="w-4 h-4 mr-2" />
Use Input
</Button>
</div>
<div class="w-full min-h-0 grow overflow-auto">
{#if Object.keys(selectedInput?.args || {}).length > 0}
{#if hasLargeArgs}
{#if largeArgs}
<div class=" overflow-auto h-full p-2">
<ObjectViewer json={largeArgs} />
</div>
{:else}
<Skeleton layout={[[8]]} />
{/if}
{:else if Object.keys(selectedInput?.args || {}).length > 0}
<div class=" overflow-auto h-full p-2">
<ObjectViewer json={selectedInput?.args} />
</div>
frontend/src/lib/components/TestJobLoader.svelte
+16 -5
View File
@@ -97,7 +97,11 @@
running: `running` in job && job.running,
logOffset: job.logs?.length ?? 0
})
job.logs = (job.logs ?? '').concat(getUpdate.new_logs ?? '')
if ((job.logs ?? '').length == 0) {
job.logs = getUpdate.new_logs ?? ''
logOffset = getUpdate.log_offset ?? 0
}
}
}
@@ -167,19 +171,26 @@
if (currentId === id) {
try {
if (job && `running` in job) {
const offset = logOffset == 0 ? (job.logs?.length ? job.logs?.length + 1 : 0) : logOffset
let previewJobUpdates = await JobService.getJobUpdates({
workspace: workspace!,
id,
running: job.running,
logOffset: logOffset == 0 ? (job.logs?.length ? job.logs?.length + 1 : 0) : logOffset
logOffset: offset
})
if (previewJobUpdates.new_logs) {
if (offset == 0) {
job.logs = previewJobUpdates.new_logs ?? ''
} else {
job.logs = (job?.logs ?? '').concat(previewJobUpdates.new_logs)
}
}
if (previewJobUpdates.log_offset) {
logOffset = previewJobUpdates.log_offset ?? 0
}
if (previewJobUpdates.new_logs) {
job.logs = (job?.logs ?? '').concat(previewJobUpdates.new_logs)
}
if (previewJobUpdates.flow_status) {
job.flow_status = previewJobUpdates.flow_status as FlowStatus
}
frontend/src/lib/components/apps/components/buttons/AppButton.svelte
-1
View File
@@ -192,7 +192,6 @@
if (iterContext && listInputs) {
listInputs.set(id, inputOutput)
}
console.log('success', r)
}}
refreshOnStart={resolvedConfig.triggerOnAppLoad}
>

Some files were not shown because too many files have changed in this diff Show More