From f28ea9cb991bbda32ed1b7a37a5f1b3552a589a8 Mon Sep 17 00:00:00 2001 From: Ruben Fiszel Date: Wed, 8 Jul 2026 18:42:39 +0200 Subject: [PATCH] feat(db-health): add connection sizing guidance (#10014) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(db-health): add connection sizing guidance The Database Connections panel showed current/max connections but gave no guidance on how to size max_connections for the deployment. Derive an estimate from the live worker fleet: each worker instance shares a pool sized DEFAULT_MAX_CONNECTIONS_WORKER + (workers - 1), and each server opens up to DEFAULT_MAX_CONNECTIONS_SERVER (both overridable via DATABASE_CONNECTIONS). The endpoint now returns live worker/instance counts, the default per-server and per-worker pool sizes, the estimated peak worker connections, the reserved superuser connections, and a recommended max_connections floor (workers + one server + 25% headroom). Servers do not ping worker_ping, so the recommendation assumes one server and exposes the per-server increment. The panel renders this as a sizing breakdown and warns when max_connections is below the recommended floor. Fixes WIN-2147 Co-Authored-By: Claude Opus 4.8 (1M context) * refactor(db-health): single source for pool-size constants + sizing tests Address review: db_connect.rs kept its own copies of DEFAULT_MAX_CONNECTIONS_* that duplicate the windmill_common constants the sizing guidance reads, so tuning the runtime pool size would silently leave the guidance stale. Re-export the windmill_common constants from db_connect.rs so there is one source of truth. Add unit tests for compute_connection_sizing covering the zero-fleet, single worker, multi-instance, and reserved-clamp cases. Co-Authored-By: Claude Opus 4.8 (1M context) * feat(db-health): 20% headroom and 200-connection minimum floor Lower the sizing headroom from 25% to 20% and never recommend below 200 connections (postgres defaults to 100; cheap headroom for growth/bursts/psql). Update the guidance message and unit tests accordingly. Co-Authored-By: Claude Opus 4.8 (1M context) * fix(db-health): honor DATABASE_CONNECTIONS in sizing recommendation Address Codex P1: the runtime caps every process's pool at DATABASE_CONNECTIONS when set (db_connect.rs), but the sizing guidance always used the default 50/5 pools. For a tuned deployment this under-estimated worker demand and could hide a genuine under-provisioning (e.g. DATABASE_CONNECTIONS=100 with 5 instances is 500 worker connections, not 25). compute_connection_sizing now takes the effective DATABASE_CONNECTIONS override (read the same way db_connect.rs reads it): when set, each worker instance and server pool is that value and the worker estimate is override * instances. The response exposes server_pool_size / worker_pool_size (effective) and database_connections_override; the panel renders both pool rows and labels them (default) vs (DATABASE_CONNECTIONS), and the message states which source is used. Adds a unit test for the override path. Co-Authored-By: Claude Opus 4.8 (1M context) * fix(db-health): exclude agent workers from connection sizing Agent workers reach the API over HTTP (MODE=agent, Connection::Http) and hold no postgres pool, but their pings still land in worker_ping (written server-side by /api/agent_workers/update_ping). Counting them inflated the connection estimate. Filter the fleet query by the worker-name prefixes: DB-connected workers use "wk-" (WORKER_NAME_PREFIX), agent workers use "ag-" (AGENT_WORKER_NAME_PREFIX). Only wk- workers/instances feed the estimate; ag- workers are counted separately and surfaced as context ("N agent workers excluded — they use HTTP, not postgres connections"). Adds a unit test. Co-Authored-By: Claude Opus 4.8 (1M context) --------- Co-authored-by: Claude Opus 4.8 (1M context) --- ...97c9af65dd72a682d9d704a2dafb563001f83.json | 35 +++ ...6cfbf0fa45c2463b861c824fb9f1c0c46192f.json | 20 ++ backend/src/db_connect.rs | 8 +- backend/windmill-api/src/db_health.rs | 251 ++++++++++++++++++ .../instanceSettings/DbHealth.svelte | 66 +++++ 5 files changed, 377 insertions(+), 3 deletions(-) create mode 100644 backend/.sqlx/query-4556f04f9da4adffb296b9c45bd97c9af65dd72a682d9d704a2dafb563001f83.json create mode 100644 backend/.sqlx/query-80e233d7db456ec7030486b009b6cfbf0fa45c2463b861c824fb9f1c0c46192f.json diff --git a/backend/.sqlx/query-4556f04f9da4adffb296b9c45bd97c9af65dd72a682d9d704a2dafb563001f83.json b/backend/.sqlx/query-4556f04f9da4adffb296b9c45bd97c9af65dd72a682d9d704a2dafb563001f83.json new file mode 100644 index 0000000000..e70052886f --- /dev/null +++ b/backend/.sqlx/query-4556f04f9da4adffb296b9c45bd97c9af65dd72a682d9d704a2dafb563001f83.json @@ -0,0 +1,35 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT\n COUNT(*) FILTER (WHERE worker LIKE $1) as \"live_workers!\",\n COUNT(DISTINCT worker_instance) FILTER (WHERE worker LIKE $1) as \"live_instances!\",\n COUNT(*) FILTER (WHERE worker LIKE $2) as \"live_agent_workers!\"\n FROM worker_ping\n WHERE ping_at > now() - interval '30 seconds'", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "live_workers!", + "type_info": "Int8" + }, + { + "ordinal": 1, + "name": "live_instances!", + "type_info": "Int8" + }, + { + "ordinal": 2, + "name": "live_agent_workers!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + null, + null, + null + ] + }, + "hash": "4556f04f9da4adffb296b9c45bd97c9af65dd72a682d9d704a2dafb563001f83" +} diff --git a/backend/.sqlx/query-80e233d7db456ec7030486b009b6cfbf0fa45c2463b861c824fb9f1c0c46192f.json b/backend/.sqlx/query-80e233d7db456ec7030486b009b6cfbf0fa45c2463b861c824fb9f1c0c46192f.json new file mode 100644 index 0000000000..bdba1e899a --- /dev/null +++ b/backend/.sqlx/query-80e233d7db456ec7030486b009b6cfbf0fa45c2463b861c824fb9f1c0c46192f.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT setting::bigint as \"v!\" FROM pg_settings WHERE name = 'superuser_reserved_connections'", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "v!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "80e233d7db456ec7030486b009b6cfbf0fa45c2463b861c824fb9f1c0c46192f" +} diff --git a/backend/src/db_connect.rs b/backend/src/db_connect.rs index deda032058..d4dee19f82 100644 --- a/backend/src/db_connect.rs +++ b/backend/src/db_connect.rs @@ -3,9 +3,11 @@ use windmill_common::{ get_database_url, DatabaseUrl, }; -pub const DEFAULT_MAX_CONNECTIONS_SERVER: u32 = 50; -pub const DEFAULT_MAX_CONNECTIONS_WORKER: u32 = 5; -pub const DEFAULT_MAX_CONNECTIONS_INDEXER: u32 = 5; +// Single source of truth in windmill_common so the DB-health sizing guidance +// (windmill-api/src/db_health.rs) and the actual pool sizing here can't drift. +pub use windmill_common::{ + DEFAULT_MAX_CONNECTIONS_INDEXER, DEFAULT_MAX_CONNECTIONS_SERVER, DEFAULT_MAX_CONNECTIONS_WORKER, +}; #[cfg(feature = "operator")] pub const DEFAULT_MAX_CONNECTIONS_OPERATOR: u32 = 2; diff --git a/backend/windmill-api/src/db_health.rs b/backend/windmill-api/src/db_health.rs index 412c126dec..c487d29da9 100644 --- a/backend/windmill-api/src/db_health.rs +++ b/backend/windmill-api/src/db_health.rs @@ -96,8 +96,35 @@ pub struct ConnectionPoolInfo { pub pg_total_connections: i64, pub pg_active_connections: i64, pub pg_idle_connections: i64, + pub pg_superuser_reserved_connections: i64, pub status: HealthLevel, pub message: String, + /// Connection sizing guidance derived from the live Windmill fleet. + pub sizing: ConnectionSizingInfo, +} + +#[derive(Serialize)] +pub struct ConnectionSizingInfo { + /// Live DB-connected worker processes (distinct worker_instance pinged recently). + pub live_worker_instances: i64, + /// Live individual DB-connected workers across all instances. + pub live_workers: i64, + /// Live agent workers (HTTP-only, hold no postgres connections; excluded from the estimate). + pub live_agent_workers: i64, + /// Effective per-server pool ceiling: DATABASE_CONNECTIONS if set, else DEFAULT_MAX_CONNECTIONS_SERVER. + pub server_pool_size: i64, + /// Effective per-worker-instance pool ceiling (single-worker baseline; grows +1 per extra worker unless DATABASE_CONNECTIONS is set). + pub worker_pool_size: i64, + /// The DATABASE_CONNECTIONS override, if this server has one set (caps every process's pool). + pub database_connections_override: Option, + /// Estimated peak connections opened by all live worker instances. + pub estimated_worker_connections: i64, + /// Recommended max_connections floor (workers + one server + headroom). + pub recommended_max_connections: i64, + /// Per-additional-server increment to add to the recommendation. + pub per_server_increment: i64, + /// Human-readable sizing explanation. + pub message: String, } #[derive(Serialize)] @@ -379,6 +406,13 @@ async fn fetch_connection_pool(db: &DB) -> windmill_common::error::Result windmill_common::error::Result now() - interval '30 seconds'"#, + db_worker_pattern, + agent_worker_pattern, + ) + .fetch_one(db) + .await?; + + // Matches how db_connect.rs reads it: when DATABASE_CONNECTIONS is set it caps + // every process's pool (server, indexer, worker) regardless of worker count. + let database_connections_override = std::env::var("DATABASE_CONNECTIONS") + .ok() + .and_then(|n| n.parse::().ok()) + .filter(|n| *n > 0); + + let sizing = compute_connection_sizing( + fleet.live_workers, + fleet.live_instances, + fleet.live_agent_workers, + reserved, + database_connections_override, + ); + let pg_max = max_row; let pg_total = stats_row.total; let pg_active = stats_row.active; @@ -438,11 +507,105 @@ async fn fetch_connection_pool(db: &DB) -> windmill_common::error::Result, +) -> ConnectionSizingInfo { + // Never recommend below this floor: postgres defaults to 100 and headroom + // for growth/bursts/psql is cheap, so 200 is a safe baseline for any fleet. + const MIN_RECOMMENDED_MAX_CONNECTIONS: i64 = 200; + + let default_server_pool = windmill_common::DEFAULT_MAX_CONNECTIONS_SERVER as i64; + let default_worker_pool = windmill_common::DEFAULT_MAX_CONNECTIONS_WORKER as i64; + + let (server_pool, worker_pool_size, estimated_worker_connections) = + match database_connections_override { + // Override caps every process identically; per-instance pool is the override. + Some(n) => (n, n, n * live_instances), + None => ( + default_server_pool, + default_worker_pool, + (default_worker_pool - 1) * live_instances + live_workers, + ), + }; + + // Workers + one server, plus 20% headroom and the superuser reserve, so the + // recommendation leaves room for psql/monitoring sessions and bursts, then + // floored at MIN_RECOMMENDED_MAX_CONNECTIONS. + let base = estimated_worker_connections + server_pool; + let recommended = ((((base as f64) * 1.20).ceil() as i64) + reserved.max(3)) + .max(MIN_RECOMMENDED_MAX_CONNECTIONS); + + let pool_source = if database_connections_override.is_some() { + format!("DATABASE_CONNECTIONS={server_pool}") + } else { + "defaults, configurable via DATABASE_CONNECTIONS".to_string() + }; + + let message = if live_instances == 0 { + format!( + "No live workers detected. Each Windmill server and worker instance opens up to {server_pool} connections ({pool_source}). Size max_connections as (servers + worker instances) × {server_pool} + ~20% headroom, and at least {MIN_RECOMMENDED_MAX_CONNECTIONS}." + ) + } else { + let per_instance = if database_connections_override.is_some() { + format!("each instance up to {worker_pool_size}") + } else { + format!("each instance up to {worker_pool_size}, +1 per extra worker") + }; + let agent_note = if live_agent_workers > 0 { + format!( + " ({live_agent_workers} agent worker(s) excluded — they use HTTP, not postgres connections.)" + ) + } else { + String::new() + }; + format!( + "{live_workers} live worker(s) across {live_instances} instance(s) can open up to ~{estimated_worker_connections} connections ({per_instance}; {pool_source}). Each Windmill server adds up to {server_pool}. Recommended max_connections ≥ {recommended} for a single server; add {server_pool} per additional server.{agent_note}" + ) + }; + + ConnectionSizingInfo { + live_worker_instances: live_instances, + live_workers, + live_agent_workers, + server_pool_size: server_pool, + worker_pool_size, + database_connections_override, + estimated_worker_connections, + recommended_max_connections: recommended, + per_server_increment: server_pool, + message, + } +} + async fn fetch_table_maintenance( db: &DB, ) -> windmill_common::error::Result> { @@ -638,3 +801,91 @@ async fn fetch_datatables(db: &DB) -> windmill_common::error::Result 500 worker connections. + let s = compute_connection_sizing(20, 5, 0, 3, Some(100)); + assert_eq!(s.server_pool_size, 100); + assert_eq!(s.worker_pool_size, 100); + assert_eq!(s.database_connections_override, Some(100)); + assert_eq!(s.estimated_worker_connections, 500); + assert_eq!(s.per_server_increment, 100); + // ceil((500 + 100) * 1.20) + 3 = 720 + 3 = 723. + assert_eq!(s.recommended_max_connections, 723); + assert!(s.message.contains("DATABASE_CONNECTIONS=100")); + } + + #[test] + fn agent_workers_are_excluded_from_the_estimate() { + // 50 agent workers alongside 2 DB workers/2 instances: only the DB + // workers count toward connections; the agent count is reported. + let s = compute_connection_sizing(2, 2, 50, 3, None); + assert_eq!(s.live_agent_workers, 50); + // (5 - 1) * 2 + 2 = 10, agent workers contribute nothing. + assert_eq!(s.estimated_worker_connections, 10); + let without_agents = compute_connection_sizing(2, 2, 0, 3, None); + assert_eq!( + s.recommended_max_connections, + without_agents.recommended_max_connections + ); + assert!(s.message.contains("50 agent worker(s) excluded")); + } +} diff --git a/frontend/src/lib/components/instanceSettings/DbHealth.svelte b/frontend/src/lib/components/instanceSettings/DbHealth.svelte index b1a16bdad5..647c4390d8 100644 --- a/frontend/src/lib/components/instanceSettings/DbHealth.svelte +++ b/frontend/src/lib/components/instanceSettings/DbHealth.svelte @@ -289,6 +289,11 @@

Total connections: {data.connection_pool.pg_total_connections} / Max: {data.connection_pool.pg_max_connections} + {#if data.connection_pool.pg_superuser_reserved_connections > 0} + ({data.connection_pool.pg_superuser_reserved_connections} reserved for superuser) + {/if}

Active: {data.connection_pool.pg_active_connections} @@ -297,6 +302,67 @@

{data.connection_pool.message}

+ + {#if data.connection_pool.sizing} + {@const sizing = data.connection_pool.sizing} + {@const undersized = + sizing.recommended_max_connections > data.connection_pool.pg_max_connections} +
+

Connection sizing guidance

+
+ Live worker instances + {formatNumber(sizing.live_worker_instances)} + Live workers + {formatNumber(sizing.live_workers)} + {#if sizing.live_agent_workers > 0} + Agent workers (HTTP, excluded) + {formatNumber(sizing.live_agent_workers)} + {/if} + Est. peak worker connections + {formatNumber(sizing.estimated_worker_connections)} + Per-server pool {sizing.database_connections_override != null + ? '(DATABASE_CONNECTIONS)' + : '(default)'} + {formatNumber(sizing.server_pool_size)} + Per-worker-instance pool {sizing.database_connections_override != null + ? '(DATABASE_CONNECTIONS)' + : '(default)'} + {formatNumber(sizing.worker_pool_size)} + Recommended max_connections + ≥ {formatNumber(sizing.recommended_max_connections)} +
+

{sizing.message}

+ {#if undersized} +

+ Current max_connections ({formatNumber( + data.connection_pool.pg_max_connections + )}) is below the recommended floor for a single server. Increase max_connections + or lower per-process pools via DATABASE_CONNECTIONS. +

+ {/if} +
+ {/if} {/if}