Files
windmill/backend/windmill-trigger-postgres
hugocasa a425431e90 feat(backend): auto-reconnect postgres trigger listener with backoff (WIN-2073) (#9666)
* feat(backend): auto-reconnect postgres trigger listener with backoff

The Postgres trigger listener permanently disabled itself on any
connection error (stream close, receive error), so a transient network
interruption (e.g. a cloud provider maintenance window) permanently
killed the trigger.

Restructure the listener to match the Kafka trigger: the replication
connection is now established inside an outer reconnect loop in
`consume`. On a dropped stream or receive error it backs off 30s and
reconnects instead of disabling, reporting a critical error every 10
failed attempts and a recovery once it reconnects. Disabling is kept
only for unrecoverable misconfiguration (missing publication or
replication slot, unparsable replication message).

Fixes WIN-2073

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(backend): count postgres reconnects on stream drop and alert from first

Adopt the SQS listener's reconnection accounting in the Postgres trigger
listener. A stream close or receive error now counts toward the retry
counter and raises a throttled critical error (on the first occurrence,
then every 10 attempts), and the retry counter is reset / recovery is
reported only once the reconnected stream actually delivers a message.

Previously the inner-loop disconnect branches reset `tries` to 0 on every
successful (re)connect and never alerted, so a stream that connected and
then immediately dropped could ping-pong every 30s indefinitely without
ever raising an alert. Resetting on real progress rather than on a bare
connect closes that blind spot and matches the SQS pattern.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-19 09:56:36 +00:00
..