warmbly/docs/MULTI_IP_WORKERS.md

Multi-IP Worker Deployment

This runbook covers the hosted Warmbly deployment pattern where one physical box runs many worker processes, one per IP. The intent is to spread sending across many sender identities (one worker per IP) without exploding the number of VMs we operate.

Why one worker per IP

Each row in the workers table is effectively an egress: a unique (worker_id, ip_addr) that the control plane assigns mailboxes to and routes Kafka work toward. The worker process binds outbound SMTP/IMAP/HTTPS to that IP via WORKER_BIND_IP and the internal/client/netbind helper. So:

• 1 IP per process keeps reputation per-IP measurable and isolatable
• N processes per box keeps ops density high without inflating VM count
• A bad IP can be quarantined by stopping a single instance, not the box

Reference recipe: Hetzner CX32 + 16 Primary IPs

CX32 = 4 vCPU, 8GB RAM, 80GB disk. Comfortable for 16 lightweight worker processes plus a Docker daemon. If you need more headroom, step up to CX42.

1. Order the IPs

In the Hetzner Cloud console, allocate 16 Primary IPv4 addresses in the same datacenter as the server and attach all of them to the CX32.

2. Configure them at the OS level

Hetzner only auto-configures the first Primary IP. The others must be added explicitly. With iproute2:

# eth0 is the public interface; check `ip link` to confirm.
for ip in 5.6.7.11 5.6.7.12 5.6.7.13 5.6.7.14 5.6.7.15 5.6.7.16 \
          5.6.7.17 5.6.7.18 5.6.7.19 5.6.7.20 5.6.7.21 5.6.7.22 \
          5.6.7.23 5.6.7.24 5.6.7.25 5.6.7.26; do
  ip addr add "${ip}/32" dev eth0
done

To make that survive reboot, use netplan (Ubuntu) or systemd-networkd. A netplan example:

# /etc/netplan/60-extra-ips.yaml
network:
  version: 2
  ethernets:
    eth0:
      addresses:
        - 5.6.7.11/32
        - 5.6.7.12/32
        # ...one entry per extra IP

Run netplan apply and confirm with ip -4 addr show eth0 that all 16 addresses are bound.

3. Set rDNS per IP

Reverse DNS (PTR) records help inbox providers identify and trust your sending. Set the PTR for each IP via the Hetzner API:

HCLOUD_TOKEN="..."
for ip in 5.6.7.11 5.6.7.12 5.6.7.13 5.6.7.14 5.6.7.15 5.6.7.16 \
          5.6.7.17 5.6.7.18 5.6.7.19 5.6.7.20 5.6.7.21 5.6.7.22 \
          5.6.7.23 5.6.7.24 5.6.7.25 5.6.7.26; do
  hostname="mail-$(echo "$ip" | tr . -).send.warmbly.com"
  curl -fsS -X POST "https://api.hetzner.cloud/v1/primary_ips/<PRIMARY_IP_ID>/actions/change_dns_ptr" \
    -H "Authorization: Bearer ${HCLOUD_TOKEN}" \
    -H "Content-Type: application/json" \
    -d "{\"ip\":\"${ip}\",\"dns_ptr\":\"${hostname}\"}"
done

Then add matching forward A records for those hostnames so rDNS + forward DNS agree. SPF, DKIM, and DMARC alignment for the sending domain still applies at the mailbox level, not the IP level.

4. Install the workers

One command brings up 16 worker processes, each bound to a distinct IP:

sudo ./install-worker.sh \
  --kafka kafka.warmbly.com:9092 \
  --schema-registry https://schema.warmbly.com \
  --redis redis://cache.warmbly.com:6379 \
  --aws-key AKIA... --aws-secret ... --aws-region us-east-1 \
  --ips 5.6.7.11,5.6.7.12,5.6.7.13,5.6.7.14,5.6.7.15,5.6.7.16,5.6.7.17,5.6.7.18,5.6.7.19,5.6.7.20,5.6.7.21,5.6.7.22,5.6.7.23,5.6.7.24,5.6.7.25,5.6.7.26

Under the hood:

• A warmbly-worker@.service systemd template unit is written once
• Per-instance env files land at /etc/warmbly/instances/<dashed-ip>.env each carrying WORKER_BIND_IP and the derived WORKER_ID
• 16 instances are enabled and started: warmbly-worker@5-6-7-11, warmbly-worker@5-6-7-12, ...
• All instances share /etc/warmbly/worker.env for Kafka, KMS, Redis, etc.

The worker IDs are deterministic UUIDv5 values derived from the IP, so reinstalling with the same --ips set produces the same IDs and reuses the existing workers rows / reputation. Adding or removing an IP only affects that instance; the rest keep sending.

Day-2 operations

Check status across all instances

sudo ./install-worker.sh --status

Prints one row per instance with IP, instance name, systemd state, and worker ID.

Tail one instance

journalctl -u warmbly-worker@5-6-7-11 -f

Restart one instance

sudo systemctl restart warmbly-worker@5-6-7-11

Quarantine one IP

If an IP's reputation tanks, take just that instance down. The rest of the box keeps sending and the control plane will stop assigning new work to the silent worker via the existing heartbeat-based assignment logic.

sudo systemctl stop warmbly-worker@5-6-7-11
sudo systemctl disable warmbly-worker@5-6-7-11

When the IP is recovered (rDNS fixed, mailbox health restored, etc.):

sudo systemctl enable --now warmbly-worker@5-6-7-11

Update all instances to a new image tag

sudo WARMBLY_WORKER_IMAGE=ghcr.io/warmbly/worker:v1.42.0 \
  ./install-worker.sh --update

--update detects multi-IP installs automatically and restarts every instance.

Uninstall

sudo ./install-worker.sh --uninstall

Stops every instance, removes the template unit and per-instance env files, and keeps the shared config under /etc/warmbly/ so a reinstall is fast. Use --purge to also delete the shared config.

Health-check model

Each worker process runs its own heartbeat goroutine (WorkerService.Heartbeat). The control plane sees N independent heartbeats from one box and treats each one as a distinct sending identity:

• one IP down -> one missed heartbeat -> control plane stops routing campaign/warmup work to that worker only
• box down -> all N heartbeats stop -> control plane reassigns affected mailboxes to other workers per the existing tier rules

There is no box-level health check: the unit of health is the worker process, which is also the unit of sending identity. This is intentional.

Blast radius guidance

Do not concentrate too much of your fleet on one box. If a box goes hard down, you lose every IP it was hosting at once. Rule of thumb:

• never put more than ~25% of your total fleet IPs on a single box
• spread boxes across at least two Hetzner datacenters (e.g. nbg1 + fsn1)
• keep premium tier and free tier on different boxes so a free-tier abuse incident cannot starve premium customers of capacity

For the default CX32 + 16 IP recipe, that means: don't run hosted Warmbly on fewer than 4 such boxes once you're past the pilot stage.

Compatibility

install-worker.sh without --ips is unchanged: one warmbly-worker.service unit, IP auto-detected or supplied via --ip. Existing single-IP deployments keep working with no migration needed. The control-plane schema (workers table) and assignment logic (internal/app/worker/assignment.go) are not affected by this change; each per-IP process simply registers itself as a normal worker.