starred/windmill
1
0
Fork 0
mirror of https://github.com/windmill-labs/windmill.git synced 2026-08-18 16:02:10 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat: v2 job debouncing (#7411)

Browse Source 
* rework everything again

Signed-off-by: pyranota <pyra@duck.com>

* updcate sqlx

Signed-off-by: pyranota <pyra@duck.com>

* update ref

Signed-off-by: pyranota <pyra@duck.com>

* fix things

Signed-off-by: pyranota <pyra@duck.com>

* fix function

Signed-off-by: pyranota <pyra@duck.com>

* final fixes

Signed-off-by: pyranota <pyra@duck.com>

* update sqlx

Signed-off-by: pyranota <pyra@duck.com>

* fix script creation

Signed-off-by: pyranota <pyra@duck.com>

* address todo

Signed-off-by: pyranota <pyra@duck.com>

* cleanup

Signed-off-by: pyranota <pyra@duck.com>

* remove dbg

Signed-off-by: pyranota <pyra@duck.com>

* cleanup

Signed-off-by: pyranota <pyra@duck.com>

* fix

Signed-off-by: pyranota <pyra@duck.com>

* fixups

Signed-off-by: pyranota <pyra@duck.com>

* fix cargo.toml

Signed-off-by: pyranota <pyra@duck.com>

* update ee repo

Signed-off-by: pyranota <pyra@duck.com>

* fix ci

Signed-off-by: pyranota <pyra@duck.com>

* nit

Signed-off-by: pyranota <pyra@duck.com>

* ref

Signed-off-by: pyranota <pyra@duck.com>

* fix

Signed-off-by: pyranota <pyra@duck.com>

* ee repo

Signed-off-by: pyranota <pyra@duck.com>

* sqlx

Signed-off-by: pyranota <pyra@duck.com>

* ee ref

Signed-off-by: pyranota <pyra@duck.com>

* remove dbg

Signed-off-by: pyranota <pyra@duck.com>

* sqlx

Signed-off-by: pyranota <pyra@duck.com>

* feat: v2 job debouncing

Signed-off-by: pyranota <pyra@duck.com>

* v2 debouncing

Signed-off-by: pyranota <pyra@duck.com>

* compat

Signed-off-by: pyranota <pyra@duck.com>

* fix all

Signed-off-by: pyranota <pyra@duck.com>

* remove legacy logic

Signed-off-by: pyranota <pyra@duck.com>

* nit

Signed-off-by: pyranota <pyra@duck.com>

* finish

Signed-off-by: pyranota <pyra@duck.com>

* nits

Signed-off-by: pyranota <pyra@duck.com>

* stage import

Signed-off-by: pyranota <pyra@duck.com>

* remove unused import

Signed-off-by: pyranota <pyra@duck.com>

* remove comments

Signed-off-by: pyranota <pyra@duck.com>

* ee ref

Signed-off-by: pyranota <pyra@duck.com>

* ee ref

Signed-off-by: pyranota <pyra@duck.com>

* nits

Signed-off-by: pyranota <pyra@duck.com>

* Update backend/windmill-queue/src/jobs.rs

Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>

* Update backend/windmill-common/src/runnable_settings/settings.rs

Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>

* Update frontend/src/lib/components/flows/DebounceLimit.svelte

Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>

* fix nits

Signed-off-by: pyranota <pyra@duck.com>

* force no compat in ci

Signed-off-by: pyranota <pyra@duck.com>

* better default key handling

Signed-off-by: pyranota <pyra@duck.com>

* simplify

Signed-off-by: pyranota <pyra@duck.com>

* sqlx

Signed-off-by: pyranota <pyra@duck.com>

* optimize

Signed-off-by: pyranota <pyra@duck.com>

* sqlx

Signed-off-by: pyranota <pyra@duck.com>

* fix flow step

Signed-off-by: pyranota <pyra@duck.com>

* add openapi

Signed-off-by: pyranota <pyra@duck.com>

* fix ci

Signed-off-by: pyranota <pyra@duck.com>

* ee ref

Signed-off-by: pyranota <pyra@duck.com>

* rename WMDEBUG flag

Signed-off-by: pyranota <pyra@duck.com>

* ee ref

Signed-off-by: pyranota <pyra@duck.com>

* remove gate for is_flow_step on pull

Signed-off-by: pyranota <pyra@duck.com>

* remove dead code

Signed-off-by: pyranota <pyra@duck.com>

* nits

Signed-off-by: pyranota <pyra@duck.com>

* chore: update ee-repo-ref to 71d872d27de44227b82f75a9c0879d8428684880

This commit updates the EE repository reference after PR #369 was merged in windmill-ee-private.

Previous ee-repo-ref: 4c7642dc9f7f4467fc055136bd56558b66ef2d54

New ee-repo-ref: 71d872d27de44227b82f75a9c0879d8428684880

Automated by sync-ee-ref workflow.

---------

Signed-off-by: pyranota <pyra@duck.com>
Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
This commit is contained in:
Pyra
2025-12-23 14:24:04 +01:00
committed by GitHub
parent 91a00857a8
commit 9d698dabb4
45 changed files with 895 additions and 815 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/backend-test.yml
+1
View File
@@ -85,6 +85,7 @@ jobs:
CARGO_NET_GIT_FETCH_WITH_CLI: true
WMDEBUG_FORCE_V0_WORKSPACE_DEPENDENCIES: 1
WMDEBUG_FORCE_RUNNABLE_SETTINGS_V0: 1
WMDEBUG_FORCE_NO_LEGACY_DEBOUNCING_COMPAT: 1
run: |
deno --version && bun -v && go version && python3 --version
cd windmill-duckdb-ffi-internal && ./build_dev.sh && cd ..
backend/.sqlx/query-16c96166ffa6b9aec65c6072b204b52b87e3c2f3d76e47eb173fc78721355066.json
Generated
+14
View File
@@ -0,0 +1,14 @@
{
"db_name": "PostgreSQL",
"query": "\n WITH _ AS (\n UPDATE debounce_key\n SET debounced_times = 0, -- reset debounced_times\n first_started_at = now(), -- rest\n previous_job_id = NULL\n WHERE job_id = $1\n )\n UPDATE v2_job_debounce_batch \n SET debounce_batch = nextval('debounce_batch_seq') -- move to new batch\n WHERE id = $1\n ",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Uuid"
]
},
"nullable": []
},
"hash": "16c96166ffa6b9aec65c6072b204b52b87e3c2f3d76e47eb173fc78721355066"
}
backend/.sqlx/query-3d38720e807b379645d8f3ab61c6a968143d42c3014152608f7d1b252cd8085c.json
Generated
+22
View File
@@ -0,0 +1,22 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO app_version\n (app_id, value, created_by, raw_app)\n SELECT app_id, value, created_by, raw_app\n FROM app_version WHERE id = $1\n RETURNING id",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "id",
"type_info": "Int8"
}
],
"parameters": {
"Left": [
"Int8"
]
},
"nullable": [
false
]
},
"hash": "3d38720e807b379645d8f3ab61c6a968143d42c3014152608f7d1b252cd8085c"
}
backend/.sqlx/query-44bf04fb504cd1b708657f01c8fb26e81e78807315226557f08449bf43982abb.json
Generated
-15
View File
@@ -1,15 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO debounce_key (key, job_id) VALUES ($1, $2) ON CONFLICT (key) DO UPDATE SET job_id = EXCLUDED.job_id",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"Uuid"
]
},
"nullable": []
},
"hash": "44bf04fb504cd1b708657f01c8fb26e81e78807315226557f08449bf43982abb"
}
backend/.sqlx/query-454ace9ce391725ef4f4c129cd66e4c12a5c40f512b70551958178c8b4d6c183.json
Generated
+35
View File
@@ -0,0 +1,35 @@
{
"db_name": "PostgreSQL",
"query": "\n INSERT INTO debounce_key (job_id, key)\n VALUES ($1, $2)\n ON CONFLICT (key)\n DO UPDATE SET\n previous_job_id = debounce_key.job_id,\n job_id = EXCLUDED.job_id, -- replace current job with new one \n debounced_times = debounce_key.debounced_times + 1 -- evaluated only if conflict,\n -- conflict means there is already existing value,\n -- which means overriding it will also imply adding new entry to v2_job_debounce_batch and thus debouncing the job\n -- so the counter should be incremented\n RETURNING\n debounced_times,\n first_started_at,\n previous_job_id AS job_id_to_debounce\n ",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "debounced_times",
"type_info": "Int4"
},
{
"ordinal": 1,
"name": "first_started_at",
"type_info": "Timestamptz"
},
{
"ordinal": 2,
"name": "job_id_to_debounce",
"type_info": "Uuid"
}
],
"parameters": {
"Left": [
"Uuid",
"Varchar"
]
},
"nullable": [
false,
false,
true
]
},
"hash": "454ace9ce391725ef4f4c129cd66e4c12a5c40f512b70551958178c8b4d6c183"
}
backend/.sqlx/query-4f11760f5d283728ded5533e6a0b51f49fdb7c11bf7d47b8536d607e646bd265.json
Generated
-22
View File
@@ -1,22 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "SELECT job_id FROM debounce_key WHERE key = $1 AND job_id IN (SELECT id FROM v2_job_queue) FOR UPDATE",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "job_id",
"type_info": "Uuid"
}
],
"parameters": {
"Left": [
"Text"
]
},
"nullable": [
false
]
},
"hash": "4f11760f5d283728ded5533e6a0b51f49fdb7c11bf7d47b8536d607e646bd265"
}
backend/.sqlx/query-61b37cb4db6e60c2d35f7d23db5afbe04e040a8dcd1d93afaaaa320665c8779a.json
Generated
+16
View File
@@ -0,0 +1,16 @@
{
"db_name": "PostgreSQL",
"query": "\n WITH prev_sd AS (\n DELETE FROM debounce_stale_data WHERE job_id = $1 RETURNING to_relock\n ) INSERT INTO debounce_stale_data (job_id, to_relock)\n VALUES ($2, array_cat((SELECT to_relock FROM prev_sd), $3))\n ON CONFLICT (job_id) DO UPDATE SET to_relock = EXCLUDED.to_relock\n ",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Uuid",
"Uuid",
"TextArray"
]
},
"nullable": []
},
"hash": "61b37cb4db6e60c2d35f7d23db5afbe04e040a8dcd1d93afaaaa320665c8779a"
}
backend/.sqlx/query-76774e6f72c8c8b7473487e4176dc17b17372b7292e39d3888a93ff4fe49e4f5.json
Generated
-14
View File
@@ -1,14 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "DELETE FROM debounce_key WHERE key = $1",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text"
]
},
"nullable": []
},
"hash": "76774e6f72c8c8b7473487e4176dc17b17372b7292e39d3888a93ff4fe49e4f5"
}
backend/.sqlx/query-7ec724b84479c2f737637e91b8cbed6cae29f361167deee879b8b683ad1bf684.json
Generated
-15
View File
@@ -1,15 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "\n INSERT INTO debounce_stale_data (job_id, to_relock)\n VALUES ($1, $2)\n ON CONFLICT (job_id)\n DO UPDATE SET to_relock = (\n SELECT array_agg(DISTINCT x)\n FROM unnest(\n -- Combine existing array with new values, removing duplicates\n array_cat(debounce_stale_data.to_relock, EXCLUDED.to_relock)\n ) AS x\n )\n ",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Uuid",
"TextArray"
]
},
"nullable": []
},
"hash": "7ec724b84479c2f737637e91b8cbed6cae29f361167deee879b8b683ad1bf684"
}
backend/.sqlx/query-83232f2db5eb1b6fef744998e60420ef920d472286cf4c1f78452446a4bcb604.json
Generated
-22
View File
@@ -1,22 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO app_version\n (app_id, value, created_by, raw_app)\n SELECT app_id, value, created_by, raw_app\n FROM app_version WHERE id = $1\n RETURNING id",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "id",
"type_info": "Int8"
}
],
"parameters": {
"Left": [
"Int8"
]
},
"nullable": [
false
]
},
"hash": "83232f2db5eb1b6fef744998e60420ef920d472286cf4c1f78452446a4bcb604"
}
backend/.sqlx/query-8360ab72d60f07dde6ecae599e6531b5b86862029ab51fdbdd44ec16239108e2.json
Generated
+15
View File
@@ -0,0 +1,15 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO v2_job_debounce_batch (id, debounce_batch)\n -- if it the first one, nextval will be evaluated, otherwise take from the job we will debounce\n SELECT\n $2,\n COALESCE(\n (\n SELECT debounce_batch\n FROM v2_job_debounce_batch\n WHERE id = $1\n LIMIT 1\n ), -- maybe use current batch\n nextval('debounce_batch_seq')\n )\n ",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Uuid",
"Uuid"
]
},
"nullable": []
},
"hash": "8360ab72d60f07dde6ecae599e6531b5b86862029ab51fdbdd44ec16239108e2"
}
backend/.sqlx/query-a6a973dcd92d2e40fd9a1c1be42052fcd350bd47ee4f63832448b6e6f0f472f0.json
Generated
+24
View File
@@ -0,0 +1,24 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO flow_version\n (workspace_id, path, value, schema, created_by)\n\n SELECT workspace_id, path, value, schema, created_by\n FROM flow_version WHERE path = $1 AND workspace_id = $2 AND id = $3\n\n RETURNING id\n ",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "id",
"type_info": "Int8"
}
],
"parameters": {
"Left": [
"Text",
"Text",
"Int8"
]
},
"nullable": [
false
]
},
"hash": "a6a973dcd92d2e40fd9a1c1be42052fcd350bd47ee4f63832448b6e6f0f472f0"
}
backend/.sqlx/query-cc3aadd61539cfa349e65f37d04c3754d88fa8d651d8cbbd95aadfaced0c0a22.json
Generated
+26
View File
@@ -0,0 +1,26 @@
{
"db_name": "PostgreSQL",
"query": "\nDELETE FROM debounce_key\nWHERE job_id IN (SELECT id FROM v2_job_completed)\nRETURNING key,job_id\n ",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "key",
"type_info": "Varchar"
},
{
"ordinal": 1,
"name": "job_id",
"type_info": "Uuid"
}
],
"parameters": {
"Left": []
},
"nullable": [
false,
false
]
},
"hash": "cc3aadd61539cfa349e65f37d04c3754d88fa8d651d8cbbd95aadfaced0c0a22"
}
backend/.sqlx/query-cfe06702916362aaf5122bb95593eff389e0d44b7a58b69fd5c79629599902fc.json
Generated
-22
View File
@@ -1,22 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "DELETE FROM debounce_stale_data WHERE job_id = $1 RETURNING to_relock",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "to_relock",
"type_info": "TextArray"
}
],
"parameters": {
"Left": [
"Uuid"
]
},
"nullable": [
true
]
},
"hash": "cfe06702916362aaf5122bb95593eff389e0d44b7a58b69fd5c79629599902fc"
}
backend/.sqlx/query-d0e826043e5a129ae6768c274c67b6254ff6c5fd450ecdab886a3183a894d266.json
Generated
+23
View File
@@ -0,0 +1,23 @@
{
"db_name": "PostgreSQL",
"query": "WITH ids AS (\n SELECT id as job_id FROM v2_job_debounce_batch WHERE debounce_batch = (\n SELECT debounce_batch FROM v2_job_debounce_batch WHERE id = $1\n )\n ) SELECT args->>$2 FROM ids LEFT JOIN v2_job ON v2_job.id = ids.job_id\n ",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "?column?",
"type_info": "Text"
}
],
"parameters": {
"Left": [
"Uuid",
"Text"
]
},
"nullable": [
null
]
},
"hash": "d0e826043e5a129ae6768c274c67b6254ff6c5fd450ecdab886a3183a894d266"
}
backend/.sqlx/query-e65c79d792f0e8285ea9acac54bc569f22ca7c28b205533e8ba73722bf438c94.json
Generated
+22
View File
@@ -0,0 +1,22 @@
{
"db_name": "PostgreSQL",
"query": "WITH _ AS (\n DELETE FROM debounce_key WHERE job_id = $1\n ) SELECT status = 'skipped' FROM v2_job_completed WHERE id = $1",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "?column?",
"type_info": "Bool"
}
],
"parameters": {
"Left": [
"Uuid"
]
},
"nullable": [
null
]
},
"hash": "e65c79d792f0e8285ea9acac54bc569f22ca7c28b205533e8ba73722bf438c94"
}
backend/.sqlx/query-f0efa383f2025158de160577ad839ae72faf0c8fe097e6ad6d309aee9a8aede2.json
Generated
-24
View File
@@ -1,24 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO flow_version\n (workspace_id, path, value, schema, created_by)\n\n SELECT workspace_id, path, value, schema, created_by\n FROM flow_version WHERE path = $1 AND workspace_id = $2 AND id = $3\n\n RETURNING id\n ",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "id",
"type_info": "Int8"
}
],
"parameters": {
"Left": [
"Text",
"Text",
"Int8"
]
},
"nullable": [
false
]
},
"hash": "f0efa383f2025158de160577ad839ae72faf0c8fe097e6ad6d309aee9a8aede2"
}
backend/.sqlx/query-f1fe8508f6cd29d4c0d354473529fc46f506ddf98d45fc870e2855c4a4b424ea.json
Generated
-23
View File
@@ -1,23 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO debounce_key (key, job_id)\n VALUES ($1, $2)\n ON CONFLICT (key)\n DO UPDATE SET job_id = debounce_key.job_id -- No actual change, just to trigger UPDATE\n RETURNING CASE WHEN xmax != 0 THEN job_id ELSE NULL END AS job_id",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "job_id",
"type_info": "Uuid"
}
],
"parameters": {
"Left": [
"Varchar",
"Uuid"
]
},
"nullable": [
null
]
},
"hash": "f1fe8508f6cd29d4c0d354473529fc46f506ddf98d45fc870e2855c4a4b424ea"
}
backend/ee-repo-ref.txt
+1 -1
View File
@@ -1 +1 @@
576abf6519d1aa12a2b989a58a123501206284fb
71d872d27de44227b82f75a9c0879d8428684880
backend/migrations/20251216114001_v2_job_debounce_data.down.sql
+6
View File
@@ -0,0 +1,6 @@
ALTER TABLE debounce_key DROP COLUMN IF EXISTS debounced_times;
ALTER TABLE debounce_key DROP COLUMN IF EXISTS first_started_at;
ALTER TABLE debounce_key DROP COLUMN IF EXISTS previous_job_id;
DROP INDEX IF EXISTS idx_v2_job_debounce_batch_debounce_batch;
DROP TABLE IF EXISTS v2_job_debounce_batch;
DROP SEQUENCE IF EXISTS debounce_batch_seq;
backend/migrations/20251216114001_v2_job_debounce_data.up.sql
+13
View File
@@ -0,0 +1,13 @@
CREATE SEQUENCE debounce_batch_seq START 1;
CREATE TABLE IF NOT EXISTS v2_job_debounce_batch(
id UUID PRIMARY KEY,
debounce_batch BIGINT NOT NULL DEFAULT nextval('debounce_batch_seq')
);
CREATE INDEX IF NOT EXISTS idx_v2_job_debounce_batch_debounce_batch ON v2_job_debounce_batch(debounce_batch);
ALTER TABLE debounce_key ADD COLUMN IF NOT EXISTS previous_job_id UUID;
ALTER TABLE debounce_key ADD COLUMN IF NOT EXISTS first_started_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT now();
ALTER TABLE debounce_key ADD COLUMN IF NOT EXISTS debounced_times INTEGER NOT NULL DEFAULT 0;
backend/src/monitor.rs
+42
View File
@@ -1606,6 +1606,17 @@ pub async fn monitor_db(
}
};
// run every 30s (every iteration)
let cleanup_debounce_keys_completed_f = async {
if server_mode && !initial_load {
if let Some(db) = conn.as_sql() {
if let Err(e) = cleanup_debounce_keys_for_completed_jobs(&db).await {
tracing::error!("Error cleaning up debounce keys for completed jobs: {:?}", e);
}
}
}
};
// run every hour (60 minutes / 30 seconds = 120)
let cleanup_worker_group_stats_f = async {
if server_mode && iteration.is_some() && iteration.as_ref().unwrap().should_run(120) {
@@ -1726,6 +1737,7 @@ pub async fn monitor_db(
cleanup_concurrency_counters_f,
cleanup_concurrency_counters_empty_keys_f,
cleanup_debounce_keys_f,
cleanup_debounce_keys_completed_f,
cleanup_worker_group_stats_f,
);
}
@@ -2791,3 +2803,33 @@ RETURNING key,job_id
}
Ok(())
}
async fn cleanup_debounce_keys_for_completed_jobs(db: &DB) -> error::Result<()> {
// If min version doesn't support runnable settings, clean up debounce keys for completed jobs
if !*windmill_common::worker::MIN_VERSION_SUPPORTS_RUNNABLE_SETTINGS_V0.read().await {
let result = sqlx::query!(
"
DELETE FROM debounce_key
WHERE job_id IN (SELECT id FROM v2_job_completed)
RETURNING key,job_id
",
)
.fetch_all(db)
.await?;
if result.len() > 0 {
tracing::warn!(
"Cleaned up {} debounce keys for completed jobs (runnable settings v0 not supported by all workers)",
result.len()
);
for row in result {
tracing::debug!(
"Debounce key for completed job cleaned up: key: {}, job_id: {:?}",
row.key,
row.job_id
);
}
}
}
Ok(())
}
backend/tests/common/mod.rs
+3 -9
View File
@@ -143,13 +143,12 @@ impl ApiServer {
pub struct RunJob {
pub payload: JobPayload,
pub args: serde_json::Map<String, serde_json::Value>,
pub debounce_job_id_o: Option<Uuid>,
pub scheduled_for_o: Option<chrono::DateTime<chrono::Utc>>,
}
impl From<JobPayload> for RunJob {
fn from(payload: JobPayload) -> Self {
Self { payload, args: Default::default(), debounce_job_id_o: None, scheduled_for_o: None }
Self { payload, args: Default::default(), scheduled_for_o: None }
}
}
@@ -159,11 +158,6 @@ impl RunJob {
self
}
pub fn push_arg_debounce_job_id_o(mut self, job_id: Option<Uuid>) -> Self {
self.debounce_job_id_o = job_id;
self
}
pub fn push_arg_scheduled_for_o(
mut self,
scheduled_for_o: Option<chrono::DateTime<chrono::Utc>>,
@@ -173,7 +167,7 @@ impl RunJob {
}
pub async fn push(self, db: &Pool<Postgres>) -> Uuid {
let RunJob { payload, args, debounce_job_id_o, scheduled_for_o } = self;
let RunJob { payload, args, scheduled_for_o } = self;
let mut hm_args = std::collections::HashMap::new();
for (k, v) in args {
hm_args.insert(k, windmill_common::worker::to_raw_value(&v));
@@ -207,7 +201,6 @@ impl RunJob {
None,
false,
None,
debounce_job_id_o,
None,
None,
)
@@ -591,6 +584,7 @@ pub async fn assert_lockfile(
hash: ScriptHash(script.hash),
dedicated_worker: None,
language,
debouncing_settings: Default::default(),
})
.push(&db2)
.await;
backend/tests/job_payload.rs
+7
View File
@@ -131,6 +131,7 @@ mod job_payload {
path: "f/system/hello_with_nodes_flow".to_string(),
dedicated_worker: None,
version: 1443253234253454,
debouncing_settings: Default::default(),
})
.run_until_complete(&db, false, port)
.await
@@ -221,6 +222,7 @@ mod job_payload {
path: "f/system/hello_with_nodes_flow".to_string(),
dedicated_worker: None,
version: 1443253234253454,
debouncing_settings: Default::default(),
})
.run_until_complete(&db, false, port)
.await
@@ -263,6 +265,7 @@ mod job_payload {
path: "f/system/hello".to_string(),
hash: ScriptHash(123412),
language: ScriptLang::Deno,
debouncing_settings: Default::default(),
dedicated_worker: None,
})
.run_until_complete(&db, false, port)
@@ -308,6 +311,7 @@ mod job_payload {
path: "f/system/hello_with_nodes_flow".to_string(),
dedicated_worker: None,
version: 1443253234253454,
debouncing_settings: Default::default(),
})
.run_until_complete(&db, false, port)
.await
@@ -449,6 +453,7 @@ mod job_payload {
path: "f/system/hello_with_nodes_flow".to_string(),
dedicated_worker: None,
version: 1443253234253454,
debouncing_settings: Default::default(),
})
.run_until_complete(&db, false, port)
.await
@@ -521,6 +526,7 @@ mod job_payload {
path: "f/system/hello_with_preprocessor".to_string(),
dedicated_worker: None,
version: 1443253234253456,
debouncing_settings: Default::default(),
})
.run_until_complete(db, false, port)
.await
@@ -577,6 +583,7 @@ mod job_payload {
path: "f/system/hello_with_nodes_flow".to_string(),
dedicated_worker: None,
version: 1443253234253454,
debouncing_settings: Default::default(),
})
.run_until_complete(&db, false, port)
.await
backend/windmill-api/openapi.yaml
+16
View File
@@ -16555,6 +16555,14 @@ components:
type: string
debounce_delay_s:
type: integer
debounce_args_to_accumulate:
type: array
items:
type: string
max_total_debouncing_time:
type: integer
max_total_debounces_amount:
type: integer
cache_ttl:
type: number
dedicated_worker:
@@ -16658,6 +16666,14 @@ components:
type: string
debounce_delay_s:
type: integer
debounce_args_to_accumulate:
type: array
items:
type: string
max_total_debouncing_time:
type: integer
max_total_debounces_amount:
type: integer
visible_to_runner_only:
type: boolean
no_main_func:
backend/windmill-api/src/apps.rs
+11 -14
View File
@@ -61,7 +61,7 @@ use windmill_common::{
users::username_to_permissioned_as,
utils::{
http_get_from_hub, not_found_if_none, paginate, query_elems_from_hub, require_admin,
Pagination, RunnableKind, StripPath, WarnAfterExt,
Pagination, RunnableKind, StripPath,
},
variables::{build_crypt, build_crypt_with_key_suffix, encrypt},
worker::{to_raw_value, CLOUD_HOSTED},
@@ -1222,7 +1222,11 @@ async fn create_app_internal<'a>(
&db,
tx,
w_id,
JobPayload::AppDependencies { path: app.path.clone(), version: v_id },
JobPayload::AppDependencies {
path: app.path.clone(),
version: v_id,
debouncing_settings: Default::default(),
},
PushArgs { args: &args, extra: None },
&authed.username,
&authed.email,
@@ -1247,7 +1251,6 @@ async fn create_app_internal<'a>(
None,
None,
None,
None,
)
.await?;
tracing::info!("Pushed app dependency job {}", dependency_job_uuid);
@@ -1531,14 +1534,6 @@ async fn update_app_internal<'a>(
path.to_owned()
};
let v_id = if let Some(nvalue) = &ns.value {
// Row lock debounce key for path. We need this to make all updates of runnables sequential and predictable.
tokio::time::timeout(
core::time::Duration::from_secs(60),
windmill_common::jobs::lock_debounce_key(&w_id, &npath, &mut tx),
)
.warn_after_seconds(10)
.await??;
let app_id = sqlx::query_scalar!(
"SELECT id FROM app WHERE path = $1 AND workspace_id = $2",
npath,
@@ -1613,7 +1608,11 @@ async fn update_app_internal<'a>(
&db,
tx,
w_id,
JobPayload::AppDependencies { path: npath.clone(), version: v_id },
JobPayload::AppDependencies {
path: npath.clone(),
version: v_id,
debouncing_settings: Default::default(),
},
PushArgs { args: &args, extra: None },
&authed.username,
&authed.email,
@@ -1638,7 +1637,6 @@ async fn update_app_internal<'a>(
None,
None,
None,
None,
)
.await?;
tracing::info!("Pushed app dependency job {}", dependency_job_uuid);
@@ -1971,7 +1969,6 @@ async fn execute_component(
end_user_email,
None,
None,
None,
)
.await?;
backend/windmill-api/src/flows.rs
+21 -12
View File
@@ -33,8 +33,10 @@ use sqlx::{FromRow, Postgres, Transaction};
use windmill_audit::audit_oss::audit_log;
use windmill_audit::ActionKind;
use windmill_common::runnable_settings::RunnableSettingsTrait;
use windmill_common::utils::{query_elems_from_hub, WarnAfterExt};
use windmill_common::worker::{to_raw_value, CLOUD_HOSTED, MIN_VERSION_SUPPORTS_DEBOUNCING};
use windmill_common::utils::query_elems_from_hub;
use windmill_common::worker::{
to_raw_value, CLOUD_HOSTED, MIN_VERSION_SUPPORTS_DEBOUNCING, MIN_VERSION_SUPPORTS_DEBOUNCING_V2,
};
use windmill_common::HUB_BASE_URL;
use windmill_common::{
db::UserDB,
@@ -46,6 +48,7 @@ use windmill_common::{
utils::{http_get_from_hub, not_found_if_none, paginate, Pagination, RunnableKind, StripPath},
};
use windmill_git_sync::{handle_deployment_metadata, DeployedObject};
use windmill_queue::WMDEBUG_FORCE_NO_LEGACY_DEBOUNCING_COMPAT;
use windmill_queue::{push, schedule::push_scheduled_job, PushIsolationLevel};
use windmill_worker::scoped_dependency_map::ScopedDependencyMap;
@@ -541,6 +544,7 @@ async fn create_flow(
path: nf.path.clone(),
dedicated_worker: nf.dedicated_worker,
version: version,
debouncing_settings: Default::default(),
},
windmill_queue::PushArgs { args: &args, extra: None },
&authed.username,
@@ -566,7 +570,6 @@ async fn create_flow(
None,
None,
None,
None,
)
.await?;
@@ -968,14 +971,6 @@ async fn update_flow(
.await?;
}
// Row lock debounce key for path. We need this to make all updates of runnables sequential and predictable.
tokio::time::timeout(
core::time::Duration::from_secs(60),
windmill_common::jobs::lock_debounce_key(&w_id, &nf.path, &mut tx),
)
.warn_after_seconds(10)
.await??;
// tracing::error!("Updating flow: {:?}", nf.value.get());
// This will lock anyone who is trying to iterate on flow_versions with given path and parameters.
@@ -1087,6 +1082,7 @@ async fn update_flow(
path: nf.path.clone(),
dedicated_worker: nf.dedicated_worker,
version,
debouncing_settings: Default::default(),
},
windmill_queue::PushArgs { args: &args, extra: None },
&authed.username,
@@ -1112,7 +1108,6 @@ async fn update_flow(
None,
None,
None,
None,
)
.await?;
@@ -1488,6 +1483,20 @@ async fn guard_flow_from_debounce_data(nf: &NewFlow) -> Result<()> {
"Flow debouncing configuration rejected: workers are behind minimum required version for debouncing feature"
);
Err(Error::WorkersAreBehind { feature: "Debouncing".into(), min_version: "1.566.0".into() })
} else if !*MIN_VERSION_SUPPORTS_DEBOUNCING_V2.read().await
&& !nf
.parse_flow_value()?
.debouncing_settings
.is_legacy_compatible()
&& !*WMDEBUG_FORCE_NO_LEGACY_DEBOUNCING_COMPAT
{
tracing::warn!(
"Flow debouncing configuration rejected: workers are behind minimum required version for debouncing feature"
);
Err(Error::WorkersAreBehind {
feature: "V2 Debouncing".into(),
min_version: "1.597.0".into(),
})
} else {
Ok(())
}
backend/windmill-api/src/jobs.rs
-14
View File
@@ -4199,7 +4199,6 @@ pub async fn run_flow<'c>(
push_authed.as_ref(),
false,
None,
None,
trigger,
run_query.suspended_mode,
)
@@ -4481,7 +4480,6 @@ pub async fn restart_flow(
false,
None,
None,
None,
run_query.suspended_mode,
)
.await?;
@@ -4615,7 +4613,6 @@ pub async fn push_script_job_by_path_into_queue<'c>(
push_authed.as_ref(),
false,
None,
None,
trigger,
run_query.suspended_mode,
)
@@ -4792,7 +4789,6 @@ pub async fn run_workflow_as_code(
None,
None,
None,
None,
)
.await?;
@@ -5330,7 +5326,6 @@ pub async fn run_wait_result_job_by_path_get(
false,
None,
None,
None,
run_query.suspended_mode,
)
.await?;
@@ -5476,7 +5471,6 @@ pub async fn run_wait_result_script_by_path_internal(
false,
None,
None,
None,
run_query.suspended_mode,
)
.await?;
@@ -5601,7 +5595,6 @@ pub async fn run_wait_result_script_by_hash(
false,
None,
None,
None,
run_query.suspended_mode,
)
.await?;
@@ -6077,7 +6070,6 @@ async fn run_preview_script(
None,
None,
None,
None,
)
.await?;
tx.commit().await?;
@@ -6230,7 +6222,6 @@ async fn run_bundle_preview_script(
None,
None,
None,
None,
)
.await?;
job_id = Some(uuid);
@@ -6382,7 +6373,6 @@ async fn run_dependencies_job(
None,
None,
None,
None,
)
.await?;
tx.commit().await?;
@@ -6471,7 +6461,6 @@ async fn run_flow_dependencies_job(
None,
None,
None,
None,
)
.await?;
tx.commit().await?;
@@ -6828,7 +6817,6 @@ async fn run_preview_flow_job(
None,
None,
None,
None,
)
.await?;
@@ -7026,7 +7014,6 @@ async fn run_dynamic_select(
None,
None,
None,
None,
)
.await?;
tx.commit().await?;
@@ -7169,7 +7156,6 @@ pub async fn run_job_by_hash_inner(
push_authed.as_ref(),
false,
None,
None,
trigger,
run_query.suspended_mode,
)
backend/windmill-api/src/scripts.rs
+16 -11
View File
@@ -49,7 +49,7 @@ use windmill_common::{
s3_helpers::upload_artifact_to_store,
scripts::{hash_script, ScriptRunnableSettingsHandle, ScriptRunnableSettingsInline},
utils::{paginate_without_limits, WarnAfterExt},
worker::{CLOUD_HOSTED, MIN_VERSION_SUPPORTS_DEBOUNCING},
worker::{CLOUD_HOSTED, MIN_VERSION_SUPPORTS_DEBOUNCING, MIN_VERSION_SUPPORTS_DEBOUNCING_V2},
};
use windmill_common::{
@@ -69,7 +69,9 @@ use windmill_common::{
};
use windmill_git_sync::{handle_deployment_metadata, DeployedObject};
use windmill_parser_ts::remove_pinned_imports;
use windmill_queue::{schedule::push_scheduled_job, PushIsolationLevel};
use windmill_queue::{
schedule::push_scheduled_job, PushIsolationLevel, WMDEBUG_FORCE_NO_LEGACY_DEBOUNCING_COMPAT,
};
const MAX_HASH_HISTORY_LENGTH_STORED: usize = 20;
@@ -866,14 +868,6 @@ async fn create_script_internal<'c>(
)
};
// Row lock debounce key for path. We need this to make all updates of runnables sequential and predictable.
tokio::time::timeout(
core::time::Duration::from_secs(60),
windmill_common::jobs::lock_debounce_key(&w_id, &ns.path, &mut tx),
)
.warn_after_seconds(10)
.await??;
sqlx::query!(
"INSERT INTO script (workspace_id, hash, path, parent_hashes, summary, description, \
content, created_by, schema, is_template, extra_perms, lock, language, kind, tag, \
@@ -1087,6 +1081,7 @@ async fn create_script_internal<'c>(
language: ns.language,
path: ns.path.clone(),
dedicated_worker: ns.dedicated_worker,
debouncing_settings: Default::default(),
},
windmill_queue::PushArgs::from(&args),
&authed.username,
@@ -1112,7 +1107,6 @@ async fn create_script_internal<'c>(
None,
None,
None,
None,
)
.await?;
@@ -2311,6 +2305,17 @@ async fn guard_script_from_debounce_data(ns: &NewScript) -> Result<()> {
"Script debouncing configuration rejected: workers are behind minimum required version for debouncing feature"
);
Err(Error::WorkersAreBehind { feature: "Debouncing".into(), min_version: "1.566.0".into() })
} else if !*MIN_VERSION_SUPPORTS_DEBOUNCING_V2.read().await
&& !ns.debouncing_settings.is_legacy_compatible()
&& !*WMDEBUG_FORCE_NO_LEGACY_DEBOUNCING_COMPAT
{
tracing::warn!(
"Script debouncing configuration rejected: workers are behind minimum required version for debouncing feature"
);
Err(Error::WorkersAreBehind {
feature: "V2 Debouncing".into(),
min_version: "1.597.0".into(),
})
} else {
Ok(())
}
backend/windmill-api/src/triggers/trigger_helpers.rs
-1
View File
@@ -944,7 +944,6 @@ async fn trigger_script_with_retry_and_error_handler<'c>(
push_authed.as_ref(),
false,
None,
None,
Some(trigger),
suspended_mode,
)
backend/windmill-common/src/jobs.rs
+3 -28
View File
@@ -407,6 +407,7 @@ pub enum JobPayload {
hash: ScriptHash,
language: ScriptLang,
dedicated_worker: Option<bool>,
debouncing_settings: DebouncingSettings,
},
/// Flow Dependency Job
@@ -414,12 +415,14 @@ pub enum JobPayload {
path: String,
dedicated_worker: Option<bool>,
version: i64,
debouncing_settings: DebouncingSettings,
},
/// App Dependency Job
AppDependencies {
path: String,
version: i64,
debouncing_settings: DebouncingSettings,
},
/// Flow Dependency Job, exposed with API. Requirements can be partially or fully predefined
@@ -874,34 +877,6 @@ pub async fn check_tag_available_for_workspace_internal(
return Ok(());
}
pub async fn lock_debounce_key<'c>(
w_id: &str,
runnable_path: &str,
tx: &mut sqlx::Transaction<'c, sqlx::Postgres>,
) -> error::Result<Option<Uuid>> {
if !*crate::worker::MIN_VERSION_SUPPORTS_DEBOUNCING.read().await {
tracing::warn!("Debouncing is not supported on this version of Windmill. Minimum version required for debouncing support.");
return Ok(None);
}
let key = format!("{w_id}:{runnable_path}:dependency");
tracing::debug!(
workspace_id = %w_id,
runnable_path = %runnable_path,
debounce_key = %key,
"Locking debounce_key for dependency job scheduling"
);
sqlx::query_scalar!(
"SELECT job_id FROM debounce_key WHERE key = $1 AND job_id IN (SELECT id FROM v2_job_queue) FOR UPDATE",
&key
)
.fetch_optional(&mut **tx)
.await
.map_err(error::Error::from)
}
pub struct RunInlinePreviewScriptFnParams {
pub workspace_id: String,
pub content: String,
backend/windmill-common/src/runnable_settings/settings.rs
+6
View File
@@ -179,6 +179,12 @@ impl DebouncingSettings {
..self
}
}
pub fn is_legacy_compatible(&self) -> bool {
self.max_total_debouncing_time.is_none()
&& self.max_total_debounces_amount.is_none()
&& self.debounce_args_to_accumulate.is_none()
}
}
impl ConcurrencySettings {
backend/windmill-common/src/scripts.rs
+5 -4
View File
@@ -977,9 +977,9 @@ pub async fn clone_script<'c>(
w_id: &str,
deployment_message: Option<String>,
db: &DB,
tx: &mut sqlx::Transaction<'c, sqlx::Postgres>,
) -> crate::error::Result<ClonedScript> {
let s = if let Some(s) = fetch_script_for_update(base_hash, w_id, &mut **tx).await? {
let mut tx = db.begin().await?;
let s = if let Some(s) = fetch_script_for_update(base_hash, w_id, &mut *tx).await? {
s
} else {
return Err(crate::error::Error::NotFound(format!(
@@ -1065,7 +1065,7 @@ pub async fn clone_script<'c>(
codebase, has_preprocessor, on_behalf_of_email, schema_validation, assets, debounce_key, debounce_delay_s, runnable_settings_handle
FROM script WHERE hash = $2 AND workspace_id = $3;
", new_hash, base_hash.0, w_id).execute(&mut **tx).await?;
", new_hash, base_hash.0, w_id).execute(&mut *tx).await?;
// Archive base.
sqlx::query!(
@@ -1073,8 +1073,9 @@ pub async fn clone_script<'c>(
*base_hash,
w_id
)
.execute(&mut **tx)
.execute(&mut *tx)
.await?;
tx.commit().await?;
Ok(ClonedScript { old_script: ns, new_hash })
}
backend/windmill-common/src/worker.rs
+2 -4
View File
@@ -265,6 +265,7 @@ lazy_static::lazy_static! {
.unwrap_or(false);
pub static ref MIN_VERSION: Arc<RwLock<Version>> = Arc::new(RwLock::new(Version::new(0, 0, 0)));
pub static ref MIN_VERSION_SUPPORTS_DEBOUNCING_V2: Arc<RwLock<bool>> = Arc::new(RwLock::new(false));
pub static ref MIN_VERSION_SUPPORTS_RUNNABLE_SETTINGS_V0: Arc<RwLock<bool>> = Arc::new(RwLock::new(false));
/// Global flag indicating if all workers support workspace dependencies feature (>= 1.583.0)
/// This flag is updated during worker initialization by checking the minimum version across all workers
@@ -429,10 +430,6 @@ fn format_pull_query(peek: String) -> String {
raw_flow, script_entrypoint_override, preprocessed
FROM v2_job
WHERE id = (SELECT id FROM peek)
), delete_debounce AS NOT MATERIALIZED (
DELETE FROM debounce_key
USING j
WHERE j.kind::text != 'flowdependencies' AND j.kind::text != 'appdependencies' AND j.kind::text != 'dependencies' AND debounce_key.job_id = j.id
) SELECT j.id, j.workspace_id, j.parent_job, j.created_by, started_at, scheduled_for,
j.runnable_id, j.runnable_path, j.args, canceled_by,
canceled_reason, j.kind, j.trigger, j.trigger_kind, j.permissioned_as,
@@ -1298,6 +1295,7 @@ pub async fn update_min_version(conn: &Connection) -> bool {
tracing::info!("Minimal worker version: {min_version}");
}
*MIN_VERSION_SUPPORTS_DEBOUNCING_V2.write().await = min_version >= Version::new(1, 597, 0);
*MIN_VERSION_SUPPORTS_RUNNABLE_SETTINGS_V0.write().await =
min_version >= *crate::runnable_settings::MIN_VERSION_RUNNABLE_SETTINGS_V0;
backend/windmill-queue/src/jobs.rs
+390 -516
View File
File diff suppressed because it is too large Load Diff
backend/windmill-queue/src/schedule.rs
-1
View File
@@ -513,7 +513,6 @@ pub async fn push_scheduled_job<'c>(
push_authed,
false,
None,
None,
Some(TriggerMetadata::new(
Some(schedule.path.clone()),
JobTriggerKind::Schedule,
backend/windmill-worker/src/ai/tools.rs
-1
View File
@@ -459,7 +459,6 @@ async fn execute_windmill_tool(
None,
None,
None,
None,
)
.await?;
backend/windmill-worker/src/worker.rs
+16 -14
View File
@@ -986,6 +986,7 @@ pub fn start_interactive_worker_shell(
let query = ("".to_string(), make_pull_query(&[common_worker_prefix]));
#[cfg(feature = "benchmark")]
let mut bench = windmill_common::bench::BenchmarkIter::new();
let job = pull(
&db,
false,
@@ -996,22 +997,23 @@ pub fn start_interactive_worker_shell(
)
.await;
use PulledJobResultToJobErr::*;
match job {
Ok(j) => {
match j.to_pulled_job() {
Ok(j) => Ok(j
.clone()
.map(|job| NextJob::Sql { flow_runners: None, job })),
Err(PulledJobResultToJobErr::MissingConcurrencyKey(jc))
| Err(PulledJobResultToJobErr::ErrorWhilePreprocessing(jc)) => {
if let Err(err) = job_completed_tx.send_job(jc, true).await
{
tracing::error!("An error occurred while sending job completed: {:#?}", err)
}
Ok(None)
Ok(j) => match j.to_pulled_job() {
Ok(j) => Ok(j
.clone()
.map(|job| NextJob::Sql { flow_runners: None, job })),
Err(MissingConcurrencyKey(jc))
| Err(ErrorWhilePreprocessing(jc)) => {
if let Err(err) = job_completed_tx.send_job(jc, true).await {
tracing::error!(
"An error occurred while sending job completed: {:#?}",
err
)
}
Ok(None)
}
}
},
Err(err) => Err(err),
}
}
@@ -1792,7 +1794,7 @@ pub async fn run_worker(
if let Err(e) = timeout(
// Will fail if longer than 10 seconds
core::time::Duration::from_secs(10),
pulled_job_res.preprocess(db),
pulled_job_res.maybe_apply_debouncing(db),
)
.warn_after_seconds(2)
.await
backend/windmill-worker/src/worker_flow.rs
-1
View File
@@ -3412,7 +3412,6 @@ async fn push_next_flow_job(
None,
None,
None,
None,
)
.warn_after_seconds(2)
.await?;
backend/windmill-worker/src/worker_lockfiles.rs
+29 -26
View File
@@ -21,11 +21,14 @@ use windmill_common::error::Error;
use windmill_common::error::Result;
use windmill_common::flows::{FlowModule, FlowModuleValue, FlowNodeId};
use windmill_common::jobs::JobPayload;
use windmill_common::runnable_settings::DebouncingSettings;
use windmill_common::scripts::ScriptHash;
use windmill_common::utils::WarnAfterExt;
#[cfg(feature = "python")]
use windmill_common::worker::PythonAnnotations;
use windmill_common::worker::{to_raw_value, to_raw_value_owned, write_file, Connection};
use windmill_common::worker::{
to_raw_value, to_raw_value_owned, write_file, Connection, MIN_VERSION_SUPPORTS_DEBOUNCING_V2,
};
use windmill_common::workspace_dependencies::{
RawWorkspaceDependencies, WorkspaceDependencies, WorkspaceDependenciesPrefetched,
};
@@ -44,7 +47,10 @@ use windmill_git_sync::{handle_deployment_metadata, DeployedObject};
#[cfg(feature = "python")]
use windmill_parser_py_imports::parse_relative_imports;
use windmill_parser_ts::parse_expr_for_imports;
use windmill_queue::{append_logs, CanceledBy, MiniPulledJob, PushIsolationLevel};
use windmill_queue::{
append_logs, CanceledBy, MiniPulledJob, PushIsolationLevel,
WMDEBUG_FORCE_NO_LEGACY_DEBOUNCING_COMPAT,
};
// TODO: To be removed in future versions
lazy_static::lazy_static! {
@@ -497,27 +503,14 @@ pub async fn trigger_dependents_to_recompute_dependencies(
args.insert(
"triggered_by_relative_import".to_string(),
to_raw_value(&()),
to_raw_value(&true),
);
// Lock the debounce_key entry FOR UPDATE to coordinate with the push side.
// This prevents concurrent modifications during dependency job scheduling.
//
// The lock serves two purposes:
// 1. Ensures we get the current debounce_job_id atomically
// 2. Blocks new push requests from modifying this key until we commit
// 3. Blocks puller from actually starting the job and gives us a chance to still squeeze the debounce in.
//
// After our transaction commits, any pending push/pull requests can proceed with
// their debounce logic.
let debounce_job_id_o =
windmill_common::jobs::lock_debounce_key(w_id, &importer_path, &mut tx).await?;
tracing::debug!(
debounce_job_id = ?debounce_job_id_o,
importer_path = %importer_path,
"Retrieved debounce job ID (if exists)"
);
let mut debouncing_settings = DebouncingSettings {
debounce_key: Some(format!("{w_id}:{importer_path}:dependency")),
debounce_delay_s: Some(5),
..Default::default()
};
let job_payload = match importer_kind.as_str() {
// TODO: Make it query only non-archived
@@ -541,6 +534,7 @@ pub async fn trigger_dependents_to_recompute_dependencies(
hash: ScriptHash(hash),
language: info.language,
dedicated_worker: info.dedicated_worker,
debouncing_settings,
}
}
None => {
@@ -575,10 +569,13 @@ pub async fn trigger_dependents_to_recompute_dependencies(
to_raw_value(&importer_node_ids),
);
debouncing_settings.debounce_args_to_accumulate = Some(vec!["nodes_to_relock".into()]);
JobPayload::FlowDependencies {
path: importer_path.clone(),
version,
dedicated_worker: None,
debouncing_settings,
}
}
None => {
@@ -608,7 +605,9 @@ pub async fn trigger_dependents_to_recompute_dependencies(
to_raw_value(importer_node_ids),
);
JobPayload::AppDependencies { path: importer_path.clone(), version }
debouncing_settings.debounce_args_to_accumulate = Some(vec!["components_to_relock".into()]);
JobPayload::AppDependencies { path: importer_path.clone(), version, debouncing_settings }
}
None => {
ScopedDependencyMap::clear_map_for_item(importer_path, w_id, "app", tx, &None)
@@ -658,7 +657,6 @@ pub async fn trigger_dependents_to_recompute_dependencies(
None,
false,
None,
debounce_job_id_o,
None,
None,
)
@@ -1457,8 +1455,10 @@ async fn lock_modules<'c>(
}
} else {
if lock.as_ref().is_some_and(|x| !x.trim().is_empty()) {
let skip_creating_new_lock = skip_creating_new_lock(&language, &content);
if skip_creating_new_lock {
if skip_creating_new_lock(&language, &content)
&& (*MIN_VERSION_SUPPORTS_DEBOUNCING_V2.read().await
|| *WMDEBUG_FORCE_NO_LEGACY_DEBOUNCING_COMPAT)
{
tx = dependency_map
.patch(get_references(), e.id.clone(), tx)
.await?;
@@ -1978,7 +1978,10 @@ async fn lock_modules_app(
.get("lock")
.is_some_and(|x| !x.as_str().unwrap().trim().is_empty())
{
if skip_creating_new_lock(&language, &content) {
if skip_creating_new_lock(&language, &content)
&& (*MIN_VERSION_SUPPORTS_DEBOUNCING_V2.read().await
|| *WMDEBUG_FORCE_NO_LEGACY_DEBOUNCING_COMPAT)
{
dependency_map
.patch(
referenced_paths.clone(),
frontend/src/lib/components/ScriptBuilder.svelte
+16
View File
@@ -522,6 +522,12 @@
concurrency_time_window_s: script.concurrency_time_window_s,
debounce_key: emptyString(script.debounce_key) ? undefined : script.debounce_key,
debounce_delay_s: script.debounce_delay_s,
debounce_args_to_accumulate:
script.debounce_args_to_accumulate && script.debounce_args_to_accumulate.length > 0
? script.debounce_args_to_accumulate
: undefined,
max_total_debouncing_time: script.max_total_debouncing_time,
max_total_debounces_amount: script.max_total_debounces_amount,
cache_ttl: script.cache_ttl,
cache_ignore_s3_path: script.cache_ignore_s3_path,
ws_error_handler_muted: script.ws_error_handler_muted,
@@ -666,6 +672,12 @@
concurrency_time_window_s: script.concurrency_time_window_s,
debounce_key: emptyString(script.debounce_key) ? undefined : script.debounce_key,
debounce_delay_s: script.debounce_delay_s,
debounce_args_to_accumulate:
script.debounce_args_to_accumulate && script.debounce_args_to_accumulate.length > 0
? script.debounce_args_to_accumulate
: undefined,
max_total_debouncing_time: script.max_total_debouncing_time,
max_total_debounces_amount: script.max_total_debounces_amount,
cache_ttl: script.cache_ttl,
cache_ignore_s3_path: script.cache_ignore_s3_path,
ws_error_handler_muted: script.ws_error_handler_muted,
@@ -1369,6 +1381,10 @@
size="sm"
bind:debounce_delay_s={script.debounce_delay_s}
bind:debounce_key={script.debounce_key}
bind:debounce_args_to_accumulate={script.debounce_args_to_accumulate}
bind:max_total_debouncing_time={script.max_total_debouncing_time}
bind:max_total_debounces_amount={script.max_total_debounces_amount}
schema={script.schema}
placeholder={`$workspace/script/${script.path}-$args[foo]`}
/>
frontend/src/lib/components/flows/DebounceLimit.svelte
+77
View File
@@ -4,10 +4,15 @@
import { SecondsInput } from '$lib/components/common'
import { enterpriseLicense } from '$lib/stores'
import Label from '../Label.svelte'
import type { Schema } from '$lib/common'
let {
debounce_delay_s = $bindable(),
debounce_key = $bindable(),
debounce_args_to_accumulate = $bindable(),
max_total_debouncing_time = $bindable(),
max_total_debounces_amount = $bindable(),
schema = undefined,
placeholder,
size = 'xs',
color = undefined,
@@ -15,11 +20,45 @@
}: {
debounce_delay_s: number | undefined
debounce_key: string | undefined
debounce_args_to_accumulate: string[] | undefined
max_total_debouncing_time: number | undefined
max_total_debounces_amount: number | undefined
schema?: Schema
placeholder: string
size: 'xs' | 'sm'
color?: 'nord' | undefined
fontClass?: string
} = $props()
// Get list of array-type arguments from schema
let arrayArgs = $derived(
schema?.properties
? Object.entries(schema.properties)
.filter(([_, prop]) => prop.type === 'array')
.map(([key, _]) => key)
: []
)
// Single selected arg for UI (convert between single selection and array)
let selectedArg = $state(debounce_args_to_accumulate?.[0] ?? '')
$effect(() => {
if (selectedArg && selectedArg !== '') {
debounce_args_to_accumulate = [selectedArg]
} else {
debounce_args_to_accumulate = undefined
}
})
// Convert 0 to undefined for max values
$effect(() => {
if (max_total_debouncing_time === 0) {
max_total_debouncing_time = undefined
}
if (max_total_debounces_amount === 0) {
max_total_debounces_amount = undefined
}
})
</script>
<div>
@@ -69,6 +108,44 @@
{placeholder}
/>
</Label>
<Label label="Argument to accumulate (optional)">
{#snippet header()}
<Tooltip>
Select a list-type argument to accumulate across debounced executions. Values from
each debounced execution will be appended together.</Tooltip
>
{/snippet}
<select disabled={!$enterpriseLicense} bind:value={selectedArg}>
<option value="">None</option>
{#each arrayArgs as arg}
<option value={arg}>{arg}</option>
{/each}
</select>
{#if arrayArgs.length === 0}
<div class="text-xs text-gray-500 mt-1">
No list-type arguments found in schema. Define arguments with type "array" to enable
accumulation.
</div>
{/if}
</Label>
<Label label="Max total debouncing time (optional)">
{#snippet header()}
<Tooltip>
Maximum total time (in seconds) that a job can be debounced before it must execute.
Once this time is reached, the job will run regardless of ongoing debouncing.</Tooltip
>
{/snippet}
<SecondsInput disabled={!$enterpriseLicense} bind:seconds={max_total_debouncing_time} />
</Label>
<Label label="Max total debounces amount (optional)">
{#snippet header()}
<Tooltip>
Maximum number of times a job can be debounced before it must execute. Once this
count is reached, the job will run regardless of ongoing debouncing.</Tooltip
>
{/snippet}
<input type="number" disabled={!$enterpriseLicense} bind:value={max_total_debounces_amount} min="0" />
</Label>
</div>
{/if}
</div>
frontend/src/lib/components/flows/content/FlowSettings.svelte
+4
View File
@@ -477,6 +477,10 @@
fontClass="font-medium"
bind:debounce_delay_s={flowStore.val.value.debounce_delay_s}
bind:debounce_key={flowStore.val.value.debounce_key}
bind:debounce_args_to_accumulate={flowStore.val.value.debounce_args_to_accumulate}
bind:max_total_debouncing_time={flowStore.val.value.max_total_debouncing_time}
bind:max_total_debounces_amount={flowStore.val.value.max_total_debounces_amount}
schema={flowStore.val.schema}
placeholder={`$workspace/flow/${$pathStore}-$args[foo]`}
/>
{/if}
frontend/src/lib/components/runs/RunRow.svelte
+2 -1
View File
@@ -26,7 +26,8 @@
ListFilterPlus,
Play,
ShieldQuestion,
X
X,
Layers
} from 'lucide-svelte'
import { createEventDispatcher } from 'svelte'
import TimeAgo from '../TimeAgo.svelte'
openflow.openapi.yaml
+11
View File
@@ -72,6 +72,17 @@ components:
debounce_key:
type: string
description: Expression to group debounced executions
debounce_args_to_accumulate:
type: array
description: Arguments to accumulate across debounced executions
items:
type: string
max_total_debouncing_time:
type: number
description: Maximum total time in seconds that a job can be debounced
max_total_debounces_amount:
type: number
description: Maximum number of times a job can be debounced
skip_expr:
type: string
description: JavaScript expression to conditionally skip the entire flow