mirror of
https://github.com/warmbly/warmbly.git
synced 2026-08-19 08:01:16 +00:00
Matthew Meszaros
f9c02bba6e
Root cause for the 10-min auto-logout (confirmed via pg_stat_activity):
the postgres pool MaxConns was 4, and four repository functions opened
a tx without committing or rolling back. After four calls each leaked
a connection in "idle in transaction" state. Once all four were gone
the pool was permanently exhausted — every new request that needed a
connection blocked until the client gave up. The 10-min trigger is
because that's when the first /auth/refresh fires; refresh tries to
acquire a connection, hangs, eventually the browser aborts the request,
the frontend treats the failure as session expiry, kicks the user.
The four leaking sites:
- emailRepository.Search (drove the leak — Accounts page)
- campaignRepository.Search
- sequenceRepository.Create
- contactRepository.BulkUpdate
Each now has `defer tx.Rollback(ctx)` immediately after Begin, matching
the pattern used in the non-leaky sites in the same files. Rollback is
a no-op after Commit, so this is safe for both read-only tx (Search)
and read-write tx (Create / BulkUpdate).
Additional hardening so a future leak can't silently brick the backend:
- MaxConns 4 → 25. 4 was reckless even without leaks; one bursty
admin page would saturate. 25 is still well under postgres'
default max_connections=100.
- MinConns 0 → 2. Keep a couple of warm connections at idle so the
first request after a quiet period doesn't pay the connect cost.
- idle_in_transaction_session_timeout=300000 (5 min) as a session
RuntimeParam. If a code path forgets the defer, postgres aborts
the leaked tx after 5 min and reclaims the connection.
- statement_timeout=60000 (60 s) as a session RuntimeParam.
Statement runaway can't pin a connection forever.
Verified after backend restart:
SELECT count(*) FROM pg_stat_activity
WHERE datname='warmbly_dev' AND state='idle in transaction';
→ 0
69 lines
2.2 KiB
Go
69 lines
2.2 KiB
Go
package db
|
|
|
|
import (
|
|
"context"
|
|
"time"
|
|
|
|
"github.com/jackc/pgx/v5/pgxpool"
|
|
)
|
|
|
|
type DB struct {
|
|
*pgxpool.Pool
|
|
}
|
|
|
|
const (
|
|
// MaxConns sizing rationale: 4 was catastrophically low — a single
|
|
// leaked tx in a request handler is enough to deadlock the entire
|
|
// backend (e.g. /auth/refresh blocks waiting for a connection, the
|
|
// frontend treats the resulting timeout as session expiry, user is
|
|
// kicked at the 10-minute refresh boundary). 25 leaves headroom even
|
|
// under bursty admin pages while staying well under postgres'
|
|
// default max_connections=100.
|
|
defaultMaxConns = int32(25)
|
|
defaultMinConns = int32(2)
|
|
defaultMaxConnLifetime = time.Hour
|
|
defaultMaxConnIdleTime = time.Minute * 30
|
|
defaultHealthCheckPeriod = time.Minute
|
|
defaultConnectTimeout = time.Second * 5
|
|
|
|
// Postgres idle-in-transaction safety net. If a code path forgets
|
|
// `defer tx.Rollback(ctx)`, the server will abort the leaked tx
|
|
// after this many milliseconds (5 min) instead of holding the
|
|
// connection forever. Belt-and-suspenders against the bug class
|
|
// that caused the 10-minute logout.
|
|
idleInTxnTimeoutMs = "300000"
|
|
// Statement-level safety net for query runaway. 60s should be more
|
|
// than enough for any user-facing query; admin reports that need
|
|
// longer can override with SET LOCAL statement_timeout.
|
|
statementTimeoutMs = "60000"
|
|
)
|
|
|
|
func New(ctx context.Context, endpoint string) (*DB, error) {
|
|
dbConfig, err := pgxpool.ParseConfig(endpoint)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
|
|
dbConfig.MaxConns = defaultMaxConns
|
|
dbConfig.MinConns = defaultMinConns
|
|
dbConfig.MaxConnLifetime = defaultMaxConnLifetime
|
|
dbConfig.MaxConnIdleTime = defaultMaxConnIdleTime
|
|
dbConfig.HealthCheckPeriod = defaultHealthCheckPeriod
|
|
dbConfig.ConnConfig.ConnectTimeout = defaultConnectTimeout
|
|
|
|
if dbConfig.ConnConfig.RuntimeParams == nil {
|
|
dbConfig.ConnConfig.RuntimeParams = map[string]string{}
|
|
}
|
|
dbConfig.ConnConfig.RuntimeParams["idle_in_transaction_session_timeout"] = idleInTxnTimeoutMs
|
|
dbConfig.ConnConfig.RuntimeParams["statement_timeout"] = statementTimeoutMs
|
|
|
|
conn, err := pgxpool.NewWithConfig(ctx, dbConfig)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
|
|
return &DB{
|
|
Pool: conn,
|
|
}, nil
|
|
}
|