feat(db-health): add connection sizing guidance (#10014)

* feat(db-health): add connection sizing guidance

The Database Connections panel showed current/max connections but gave no
guidance on how to size max_connections for the deployment. Derive an estimate
from the live worker fleet: each worker instance shares a pool sized
DEFAULT_MAX_CONNECTIONS_WORKER + (workers - 1), and each server opens up to
DEFAULT_MAX_CONNECTIONS_SERVER (both overridable via DATABASE_CONNECTIONS).

The endpoint now returns live worker/instance counts, the default per-server
and per-worker pool sizes, the estimated peak worker connections, the reserved
superuser connections, and a recommended max_connections floor (workers + one
server + 25% headroom). Servers do not ping worker_ping, so the recommendation
assumes one server and exposes the per-server increment. The panel renders this
as a sizing breakdown and warns when max_connections is below the recommended
floor.

Fixes WIN-2147

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(db-health): single source for pool-size constants + sizing tests

Address review: db_connect.rs kept its own copies of DEFAULT_MAX_CONNECTIONS_*
that duplicate the windmill_common constants the sizing guidance reads, so
tuning the runtime pool size would silently leave the guidance stale. Re-export
the windmill_common constants from db_connect.rs so there is one source of truth.

Add unit tests for compute_connection_sizing covering the zero-fleet, single
worker, multi-instance, and reserved-clamp cases.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(db-health): 20% headroom and 200-connection minimum floor

Lower the sizing headroom from 25% to 20% and never recommend below 200
connections (postgres defaults to 100; cheap headroom for growth/bursts/psql).
Update the guidance message and unit tests accordingly.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(db-health): honor DATABASE_CONNECTIONS in sizing recommendation

Address Codex P1: the runtime caps every process's pool at DATABASE_CONNECTIONS
when set (db_connect.rs), but the sizing guidance always used the default 50/5
pools. For a tuned deployment this under-estimated worker demand and could hide
a genuine under-provisioning (e.g. DATABASE_CONNECTIONS=100 with 5 instances is
500 worker connections, not 25).

compute_connection_sizing now takes the effective DATABASE_CONNECTIONS override
(read the same way db_connect.rs reads it): when set, each worker instance and
server pool is that value and the worker estimate is override * instances. The
response exposes server_pool_size / worker_pool_size (effective) and
database_connections_override; the panel renders both pool rows and labels them
(default) vs (DATABASE_CONNECTIONS), and the message states which source is used.
Adds a unit test for the override path.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(db-health): exclude agent workers from connection sizing

Agent workers reach the API over HTTP (MODE=agent, Connection::Http) and hold
no postgres pool, but their pings still land in worker_ping (written server-side
by /api/agent_workers/update_ping). Counting them inflated the connection
estimate. Filter the fleet query by the worker-name prefixes: DB-connected
workers use "wk-" (WORKER_NAME_PREFIX), agent workers use "ag-"
(AGENT_WORKER_NAME_PREFIX). Only wk- workers/instances feed the estimate; ag-
workers are counted separately and surfaced as context ("N agent workers
excluded — they use HTTP, not postgres connections"). Adds a unit test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

backend/.sqlx/query-4556f04f9da4adffb296b9c45bd97c9af65dd72a682d9d704a2dafb563001f83.json

@@ -0,0 +1,35 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT\n            COUNT(*) FILTER (WHERE worker LIKE $1) as \"live_workers!\",\n            COUNT(DISTINCT worker_instance) FILTER (WHERE worker LIKE $1) as \"live_instances!\",\n            COUNT(*) FILTER (WHERE worker LIKE $2) as \"live_agent_workers!\"\n        FROM worker_ping\n        WHERE ping_at > now() - interval '30 seconds'",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "live_workers!",
+        "type_info": "Int8"
+      },
+      {
+        "ordinal": 1,
+        "name": "live_instances!",
+        "type_info": "Int8"
+      },
+      {
+        "ordinal": 2,
+        "name": "live_agent_workers!",
+        "type_info": "Int8"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Text",
+        "Text"
+      ]
+    },
+    "nullable": [
+      null,
+      null,
+      null
+    ]
+  },
+  "hash": "4556f04f9da4adffb296b9c45bd97c9af65dd72a682d9d704a2dafb563001f83"
+}

backend/.sqlx/query-80e233d7db456ec7030486b009b6cfbf0fa45c2463b861c824fb9f1c0c46192f.json

@@ -0,0 +1,20 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT setting::bigint as \"v!\" FROM pg_settings WHERE name = 'superuser_reserved_connections'",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "v!",
+        "type_info": "Int8"
+      }
+    ],
+    "parameters": {
+      "Left": []
+    },
+    "nullable": [
+      null
+    ]
+  },
+  "hash": "80e233d7db456ec7030486b009b6cfbf0fa45c2463b861c824fb9f1c0c46192f"
+}

backend/src/db_connect.rs

@@ -3,9 +3,11 @@ use windmill_common::{
     get_database_url, DatabaseUrl,
 };
 
-pub const DEFAULT_MAX_CONNECTIONS_SERVER: u32 = 50;
-pub const DEFAULT_MAX_CONNECTIONS_WORKER: u32 = 5;
-pub const DEFAULT_MAX_CONNECTIONS_INDEXER: u32 = 5;
+// Single source of truth in windmill_common so the DB-health sizing guidance
+// (windmill-api/src/db_health.rs) and the actual pool sizing here can't drift.
+pub use windmill_common::{
+    DEFAULT_MAX_CONNECTIONS_INDEXER, DEFAULT_MAX_CONNECTIONS_SERVER, DEFAULT_MAX_CONNECTIONS_WORKER,
+};
 #[cfg(feature = "operator")]
 pub const DEFAULT_MAX_CONNECTIONS_OPERATOR: u32 = 2;
 

backend/windmill-api/src/db_health.rs

@@ -96,8 +96,35 @@ pub struct ConnectionPoolInfo {
     pub pg_total_connections: i64,
     pub pg_active_connections: i64,
     pub pg_idle_connections: i64,
+    pub pg_superuser_reserved_connections: i64,
     pub status: HealthLevel,
     pub message: String,
+    /// Connection sizing guidance derived from the live Windmill fleet.
+    pub sizing: ConnectionSizingInfo,
+}
+
+#[derive(Serialize)]
+pub struct ConnectionSizingInfo {
+    /// Live DB-connected worker processes (distinct worker_instance pinged recently).
+    pub live_worker_instances: i64,
+    /// Live individual DB-connected workers across all instances.
+    pub live_workers: i64,
+    /// Live agent workers (HTTP-only, hold no postgres connections; excluded from the estimate).
+    pub live_agent_workers: i64,
+    /// Effective per-server pool ceiling: DATABASE_CONNECTIONS if set, else DEFAULT_MAX_CONNECTIONS_SERVER.
+    pub server_pool_size: i64,
+    /// Effective per-worker-instance pool ceiling (single-worker baseline; grows +1 per extra worker unless DATABASE_CONNECTIONS is set).
+    pub worker_pool_size: i64,
+    /// The DATABASE_CONNECTIONS override, if this server has one set (caps every process's pool).
+    pub database_connections_override: Option<i64>,
+    /// Estimated peak connections opened by all live worker instances.
+    pub estimated_worker_connections: i64,
+    /// Recommended max_connections floor (workers + one server + headroom).
+    pub recommended_max_connections: i64,
+    /// Per-additional-server increment to add to the recommendation.
+    pub per_server_increment: i64,
+    /// Human-readable sizing explanation.
+    pub message: String,
 }
 
 #[derive(Serialize)]
@@ -379,6 +406,13 @@ async fn fetch_connection_pool(db: &DB) -> windmill_common::error::Result<Connec
     .fetch_one(db)
     .await?;
 
+    let reserved = sqlx::query_scalar!(
+        r#"SELECT setting::bigint as "v!" FROM pg_settings WHERE name = 'superuser_reserved_connections'"#
+    )
+    .fetch_one(db)
+    .await
+    .unwrap_or(0);
+
     let stats_row = sqlx::query!(
         r#"SELECT
             COUNT(*) as "total!",
@@ -390,6 +424,41 @@ async fn fetch_connection_pool(db: &DB) -> windmill_common::error::Result<Connec
     .fetch_one(db)
     .await?;
 
+    // Live Windmill worker fleet: each worker pings worker_ping every ~5s; a
+    // window of 30s tolerates a missed ping without counting dead workers.
+    // Agent workers (name prefix "ag-") talk to the API over HTTP and hold no
+    // postgres pool, so they're excluded from the connection estimate and only
+    // reported for context; regular DB-connected workers use the "wk-" prefix.
+    let db_worker_pattern = format!("{}-%", windmill_common::utils::WORKER_NAME_PREFIX);
+    let agent_worker_pattern = format!("{}-%", windmill_common::utils::AGENT_WORKER_NAME_PREFIX);
+    let fleet = sqlx::query!(
+        r#"SELECT
+            COUNT(*) FILTER (WHERE worker LIKE $1) as "live_workers!",
+            COUNT(DISTINCT worker_instance) FILTER (WHERE worker LIKE $1) as "live_instances!",
+            COUNT(*) FILTER (WHERE worker LIKE $2) as "live_agent_workers!"
+        FROM worker_ping
+        WHERE ping_at > now() - interval '30 seconds'"#,
+        db_worker_pattern,
+        agent_worker_pattern,
+    )
+    .fetch_one(db)
+    .await?;
+
+    // Matches how db_connect.rs reads it: when DATABASE_CONNECTIONS is set it caps
+    // every process's pool (server, indexer, worker) regardless of worker count.
+    let database_connections_override = std::env::var("DATABASE_CONNECTIONS")
+        .ok()
+        .and_then(|n| n.parse::<i64>().ok())
+        .filter(|n| *n > 0);
+
+    let sizing = compute_connection_sizing(
+        fleet.live_workers,
+        fleet.live_instances,
+        fleet.live_agent_workers,
+        reserved,
+        database_connections_override,
+    );
+
     let pg_max = max_row;
     let pg_total = stats_row.total;
     let pg_active = stats_row.active;
@@ -438,11 +507,105 @@ async fn fetch_connection_pool(db: &DB) -> windmill_common::error::Result<Connec
         pg_total_connections: pg_total,
         pg_active_connections: pg_active,
         pg_idle_connections: pg_idle,
+        pg_superuser_reserved_connections: reserved,
         status,
         message,
+        sizing,
     })
 }
 
+/// Estimate how many postgres connections the live Windmill fleet can open and
+/// derive a recommended `max_connections` floor.
+///
+/// Pool sizing mirrors `db_connect.rs`: when `DATABASE_CONNECTIONS` is set it
+/// caps *every* process's pool (server, indexer, worker) at that value, so each
+/// worker instance opens up to that many connections. Otherwise each worker
+/// instance shares a pool of `DEFAULT_MAX_CONNECTIONS_WORKER + (workers - 1)`
+/// (fleet ceiling `(worker_pool - 1) * instances + workers`) and each server
+/// opens up to `DEFAULT_MAX_CONNECTIONS_SERVER`.
+///
+/// `database_connections_override` is this server's `DATABASE_CONNECTIONS`, our
+/// best proxy for the fleet's config. Servers do not ping `worker_ping`, so we
+/// can't count them — the recommendation assumes one server and exposes the
+/// per-server increment so the operator can add capacity for the rest.
+///
+/// `live_agent_workers` is reported for context only: agent workers reach the
+/// API over HTTP and open no postgres connections, so they never contribute to
+/// the estimate.
+fn compute_connection_sizing(
+    live_workers: i64,
+    live_instances: i64,
+    live_agent_workers: i64,
+    reserved: i64,
+    database_connections_override: Option<i64>,
+) -> ConnectionSizingInfo {
+    // Never recommend below this floor: postgres defaults to 100 and headroom
+    // for growth/bursts/psql is cheap, so 200 is a safe baseline for any fleet.
+    const MIN_RECOMMENDED_MAX_CONNECTIONS: i64 = 200;
+
+    let default_server_pool = windmill_common::DEFAULT_MAX_CONNECTIONS_SERVER as i64;
+    let default_worker_pool = windmill_common::DEFAULT_MAX_CONNECTIONS_WORKER as i64;
+
+    let (server_pool, worker_pool_size, estimated_worker_connections) =
+        match database_connections_override {
+            // Override caps every process identically; per-instance pool is the override.
+            Some(n) => (n, n, n * live_instances),
+            None => (
+                default_server_pool,
+                default_worker_pool,
+                (default_worker_pool - 1) * live_instances + live_workers,
+            ),
+        };
+
+    // Workers + one server, plus 20% headroom and the superuser reserve, so the
+    // recommendation leaves room for psql/monitoring sessions and bursts, then
+    // floored at MIN_RECOMMENDED_MAX_CONNECTIONS.
+    let base = estimated_worker_connections + server_pool;
+    let recommended = ((((base as f64) * 1.20).ceil() as i64) + reserved.max(3))
+        .max(MIN_RECOMMENDED_MAX_CONNECTIONS);
+
+    let pool_source = if database_connections_override.is_some() {
+        format!("DATABASE_CONNECTIONS={server_pool}")
+    } else {
+        "defaults, configurable via DATABASE_CONNECTIONS".to_string()
+    };
+
+    let message = if live_instances == 0 {
+        format!(
+            "No live workers detected. Each Windmill server and worker instance opens up to {server_pool} connections ({pool_source}). Size max_connections as (servers + worker instances) × {server_pool} + ~20% headroom, and at least {MIN_RECOMMENDED_MAX_CONNECTIONS}."
+        )
+    } else {
+        let per_instance = if database_connections_override.is_some() {
+            format!("each instance up to {worker_pool_size}")
+        } else {
+            format!("each instance up to {worker_pool_size}, +1 per extra worker")
+        };
+        let agent_note = if live_agent_workers > 0 {
+            format!(
+                " ({live_agent_workers} agent worker(s) excluded — they use HTTP, not postgres connections.)"
+            )
+        } else {
+            String::new()
+        };
+        format!(
+            "{live_workers} live worker(s) across {live_instances} instance(s) can open up to ~{estimated_worker_connections} connections ({per_instance}; {pool_source}). Each Windmill server adds up to {server_pool}. Recommended max_connections ≥ {recommended} for a single server; add {server_pool} per additional server.{agent_note}"
+        )
+    };
+
+    ConnectionSizingInfo {
+        live_worker_instances: live_instances,
+        live_workers,
+        live_agent_workers,
+        server_pool_size: server_pool,
+        worker_pool_size,
+        database_connections_override,
+        estimated_worker_connections,
+        recommended_max_connections: recommended,
+        per_server_increment: server_pool,
+        message,
+    }
+}
+
 async fn fetch_table_maintenance(
     db: &DB,
 ) -> windmill_common::error::Result<Vec<TableMaintenanceInfo>> {
@@ -638,3 +801,91 @@ async fn fetch_datatables(db: &DB) -> windmill_common::error::Result<Vec<Datatab
     result.sort_by(|a, b| b.size_bytes.cmp(&a.size_bytes));
     Ok(result)
 }
+
+#[cfg(test)]
+mod tests {
+    use super::compute_connection_sizing;
+
+    #[test]
+    fn no_workers_reports_defaults_and_floors_at_200() {
+        let s = compute_connection_sizing(0, 0, 0, 3, None);
+        assert_eq!(s.live_workers, 0);
+        assert_eq!(s.live_worker_instances, 0);
+        assert_eq!(s.live_agent_workers, 0);
+        assert_eq!(s.estimated_worker_connections, 0);
+        assert_eq!(s.server_pool_size, 50);
+        assert_eq!(s.worker_pool_size, 5);
+        assert_eq!(s.database_connections_override, None);
+        assert_eq!(s.per_server_increment, 50);
+        // ceil((0 + 50) * 1.20) + 3 = 63, floored up to 200.
+        assert_eq!(s.recommended_max_connections, 200);
+        assert!(s.message.contains("No live workers"));
+    }
+
+    #[test]
+    fn single_worker_single_instance_floors_at_200() {
+        let s = compute_connection_sizing(1, 1, 0, 3, None);
+        // (5 - 1) * 1 instance + 1 worker = 5
+        assert_eq!(s.estimated_worker_connections, 5);
+        // ceil((5 + 50) * 1.20) + 3 = 66 + 3 = 69, floored up to 200.
+        assert_eq!(s.recommended_max_connections, 200);
+    }
+
+    #[test]
+    fn multi_worker_instances_sum_per_instance_pools() {
+        // Two instances, 5 workers total: pools are (4 + w_i) summed = 4*2 + 5 = 13.
+        let s = compute_connection_sizing(5, 2, 0, 3, None);
+        assert_eq!(s.estimated_worker_connections, 13);
+    }
+
+    #[test]
+    fn large_fleet_exceeds_floor_with_margin_and_reserve() {
+        // 300 workers across 10 instances: (5-1)*10 + 300 = 340 worker connections.
+        let s = compute_connection_sizing(300, 10, 0, 3, None);
+        assert_eq!(s.estimated_worker_connections, 340);
+        // ceil((340 + 50) * 1.20) + 3 = ceil(468.0) + 3 = 471, above the 200 floor.
+        assert_eq!(s.recommended_max_connections, 471);
+    }
+
+    #[test]
+    fn reserved_above_default_widens_recommendation() {
+        // Big enough fleet that the floor doesn't mask the reserved contribution.
+        let base = compute_connection_sizing(300, 10, 0, 3, None);
+        let high = compute_connection_sizing(300, 10, 0, 20, None);
+        assert_eq!(
+            high.recommended_max_connections - base.recommended_max_connections,
+            17
+        );
+    }
+
+    #[test]
+    fn database_connections_override_caps_every_process() {
+        // DATABASE_CONNECTIONS=100 means each instance opens up to 100, not the
+        // default 5-based estimate. 5 instances -> 500 worker connections.
+        let s = compute_connection_sizing(20, 5, 0, 3, Some(100));
+        assert_eq!(s.server_pool_size, 100);
+        assert_eq!(s.worker_pool_size, 100);
+        assert_eq!(s.database_connections_override, Some(100));
+        assert_eq!(s.estimated_worker_connections, 500);
+        assert_eq!(s.per_server_increment, 100);
+        // ceil((500 + 100) * 1.20) + 3 = 720 + 3 = 723.
+        assert_eq!(s.recommended_max_connections, 723);
+        assert!(s.message.contains("DATABASE_CONNECTIONS=100"));
+    }
+
+    #[test]
+    fn agent_workers_are_excluded_from_the_estimate() {
+        // 50 agent workers alongside 2 DB workers/2 instances: only the DB
+        // workers count toward connections; the agent count is reported.
+        let s = compute_connection_sizing(2, 2, 50, 3, None);
+        assert_eq!(s.live_agent_workers, 50);
+        // (5 - 1) * 2 + 2 = 10, agent workers contribute nothing.
+        assert_eq!(s.estimated_worker_connections, 10);
+        let without_agents = compute_connection_sizing(2, 2, 0, 3, None);
+        assert_eq!(
+            s.recommended_max_connections,
+            without_agents.recommended_max_connections
+        );
+        assert!(s.message.contains("50 agent worker(s) excluded"));
+    }
+}

frontend/src/lib/components/instanceSettings/DbHealth.svelte

@@ -289,6 +289,11 @@
 						<p class="text-secondary">
 							Total connections: <strong>{data.connection_pool.pg_total_connections}</strong> / Max:
 							<strong>{data.connection_pool.pg_max_connections}</strong>
+							{#if data.connection_pool.pg_superuser_reserved_connections > 0}
+								<span class="text-tertiary"
+									>({data.connection_pool.pg_superuser_reserved_connections} reserved for superuser)</span
+								>
+							{/if}
 						</p>
 						<p class="text-secondary">
 							Active: <strong>{data.connection_pool.pg_active_connections}</strong>
@@ -297,6 +302,67 @@
 						<p class="{statusColor(data.connection_pool.status)} mt-1 font-medium">
 							{data.connection_pool.message}
 						</p>
+
+						{#if data.connection_pool.sizing}
+							{@const sizing = data.connection_pool.sizing}
+							{@const undersized =
+								sizing.recommended_max_connections > data.connection_pool.pg_max_connections}
+							<div class="bg-surface-secondary mt-2 flex flex-col gap-2 rounded p-2">
+								<p class="text-secondary font-semibold">Connection sizing guidance</p>
+								<div class="grid grid-cols-2 gap-x-6 gap-y-1">
+									<span class="text-tertiary">Live worker instances</span>
+									<span class="text-primary text-right font-medium"
+										>{formatNumber(sizing.live_worker_instances)}</span
+									>
+									<span class="text-tertiary">Live workers</span>
+									<span class="text-primary text-right font-medium"
+										>{formatNumber(sizing.live_workers)}</span
+									>
+									{#if sizing.live_agent_workers > 0}
+										<span class="text-tertiary">Agent workers (HTTP, excluded)</span>
+										<span class="text-primary text-right font-medium"
+											>{formatNumber(sizing.live_agent_workers)}</span
+										>
+									{/if}
+									<span class="text-tertiary">Est. peak worker connections</span>
+									<span class="text-primary text-right font-medium"
+										>{formatNumber(sizing.estimated_worker_connections)}</span
+									>
+									<span class="text-tertiary"
+										>Per-server pool {sizing.database_connections_override != null
+											? '(DATABASE_CONNECTIONS)'
+											: '(default)'}</span
+									>
+									<span class="text-primary text-right font-medium"
+										>{formatNumber(sizing.server_pool_size)}</span
+									>
+									<span class="text-tertiary"
+										>Per-worker-instance pool {sizing.database_connections_override != null
+											? '(DATABASE_CONNECTIONS)'
+											: '(default)'}</span
+									>
+									<span class="text-primary text-right font-medium"
+										>{formatNumber(sizing.worker_pool_size)}</span
+									>
+									<span class="text-tertiary">Recommended max_connections</span>
+									<span
+										class="text-right font-semibold {undersized
+											? 'text-yellow-600 dark:text-yellow-400'
+											: 'text-green-600 dark:text-green-400'}"
+										>≥ {formatNumber(sizing.recommended_max_connections)}</span
+									>
+								</div>
+								<p class="text-tertiary leading-relaxed">{sizing.message}</p>
+								{#if undersized}
+									<p class="font-medium text-yellow-600 dark:text-yellow-400">
+										Current max_connections ({formatNumber(
+											data.connection_pool.pg_max_connections
+										)}) is below the recommended floor for a single server. Increase max_connections
+										or lower per-process pools via DATABASE_CONNECTIONS.
+									</p>
+								{/if}
+							</div>
+						{/if}
 					</div>
 				{/if}
 			</section>