feat(jobs): cap total queued jobs per workspace on cloud (#10218)

* feat(jobs): cap total queued jobs per workspace on cloud

A workspace could flood the queue with an unbounded number of jobs across
many concurrency keys and scripts (or keyless jobs), which the per-key
cap from #10197 does not bound. Add a companion instance-wide ceiling on
a workspace's total queued jobs.

check_workspace_queue_cap rejects a push once the workspace has
WORKSPACE_MAX_QUEUED_JOBS (default 20000, superadmin-configurable, 0 to
disable) jobs queued, cloud-only and runtime-gated on CLOUD_HOSTED like
the per-key cap. It runs on every push, so it applies even to premium
workspaces and catches parallel for-loop floods. Jobs already queued
still drain; only new pushes past the ceiling are rejected, so an
in-flight flow only fails to push further work while at the ceiling.

The setting loader self-gates on CLOUD_HOSTED so it is never loaded off
cloud, from initial load or a settings-change reload. The depth count is
bounded by the cap via LIMIT so a runaway backlog never costs an
unbounded scan on the push path.

* docs(jobs): note the workspace cap is a soft ceiling and the depth helper is count-only

Records the two review points as constraints: the cap does not serialize
admission (a soft ceiling by design, like the per-key cap), and
workspace_queue_depth is pub only for the test, returns a count not job
data, and leaves authorization to the caller.
This commit is contained in:
Ruben Fiszel
2026-07-20 22:09:59 +02:00
committed by GitHub
parent b070f56c5e
commit ddec2abbb3
10 changed files with 277 additions and 12 deletions
@@ -0,0 +1,17 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO v2_job_queue (id, workspace_id, scheduled_for, tag, running)\n VALUES ($1, $2, now() + ($3::bigint::text || ' s')::interval, 'other', $4)",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Uuid",
"Varchar",
"Int8",
"Bool"
]
},
"nullable": []
},
"hash": "1ce0728a6b0942fecf8ce5f4a06857a2601b10b057877c3b2795b8ea9c928c1b"
}
@@ -0,0 +1,15 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO v2_job (id, workspace_id, tag) VALUES ($1, $2, 'other')",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Uuid",
"Varchar"
]
},
"nullable": []
},
"hash": "750eb7365e2590a42d9f3c4cbbf193ea4129874c36e2c334e9e73f4b4c54ad07"
}
@@ -0,0 +1,23 @@
{
"db_name": "PostgreSQL",
"query": "SELECT count(*) FROM (\n SELECT 1 FROM v2_job_queue\n WHERE workspace_id = $1 AND running = false\n LIMIT $2\n ) s",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "count",
"type_info": "Int8"
}
],
"parameters": {
"Left": [
"Text",
"Int8"
]
},
"nullable": [
null
]
},
"hash": "d977778801cc5efdfc4051d5b77d75eda5fc20669d598f0dbbb301030e58d702"
}