mirror of
https://github.com/warmbly/warmbly.git
synced 2026-08-24 08:00:37 +00:00
Matthew Meszaros
261cc439ad
Workers are no longer curl|sh-only. Admins add and manage them from the
dashboard over SSH, with all runtime config (Kafka, Schema Registry,
Redis, AWS keys) stored encrypted via the existing KMS-envelope cipher
service.
Worker lifecycle:
1. Admin POSTs host/port/user. Backend generates an ed25519 keypair,
encrypts the private key under uuid.Nil (platform identity), and
stores the row in 'pending' state.
2. Admin pastes the returned public key into the VPS's authorized_keys.
3. Test connection — runs `true` over SSH, pins the host SHA256
fingerprint on first success (TOFU).
4. Install — backend scp's install-worker.sh + a per-worker env file
and runs it. State moves pending → provisioning → installed.
5. From then on: restart, update image, apply config, uninstall,
rotate keys, tail logs, live status, OS package update, reboot —
all dashboard buttons backed by SSH operations.
Credentials are reusable entities:
- aws_credentials: named keypair, secret encrypted at rest
- worker_profiles: bundles Kafka + Schema Registry + Redis + image +
release channel, references one AWS credentials row
- workers.profile_id links a worker to a profile; many workers can
share one profile
Saving a profile doesn't restart anything. The dashboard compares
profile.updated_at to each worker's config_applied_at and shows a
"stale config" badge; Apply rewrites /etc/warmbly/worker.env over SSH
and restarts the unit.
Auto-update on GitHub release:
- profile.release_channel ∈ {pinned, stable, dev}
- profile.auto_update toggles automatic rollout
- Trigger model is push, not poll: one check on backend boot, then
the /webhooks/github/releases endpoint (HMAC-validated with
RELEASES_WEBHOOK_SECRET) on every release event. Manual "Check now"
button as fallback.
- When a new tag resolves, the orchestrator SSHes into each assigned
worker, runs install-worker.sh --update --image <new>, which now
rewrites the systemd unit (not just `docker pull`) so the image
actually changes. workers.image_version captures the running tag
for the UI's "v1.2.3 → v1.2.4" diff.
Self-hostable: every release knob is env-driven —
RELEASES_GITHUB_REPO, RELEASES_WORKER_IMAGE_REPO,
RELEASES_WEBHOOK_SECRET, RELEASES_GITHUB_TOKEN, RELEASES_ENABLED. Set
RELEASES_ENABLED=false to disable the feature entirely.
OS-level updates and reboot are also exposed: detect apt / dnf / yum /
pacman / apk, run the right upgrade noninteractively, return the full
output and a reboot-required flag. Reboots are never automatic.
Migrations:
000028_worker_ssh — ssh fields, install_state enum, last_seen,
host fingerprint
000029_worker_credentials — aws_credentials + worker_profiles +
workers.profile_id + workers.config_applied_at
000030_worker_releases — release_channel enum, auto_update,
resolved_image_tag, workers.image_version
Endpoints added:
POST /admin/workers (create + keypair)
GET /admin/workers/managed
GET /admin/workers/:id/managed
POST /admin/workers/:id/{test,install,restart,upgrade,uninstall,rotate-keys,apply,system-update,reboot}
PUT /admin/workers/:id/profile
GET /admin/workers/:id/{live-status,logs}
DELETE /admin/workers/:id
GET /admin/aws-credentials CRUD
GET /admin/worker-profiles CRUD + /workers + /apply + /release
GET /admin/releases/state
POST /admin/releases/check
POST /webhooks/github/releases public, HMAC-validated
Admin UI:
/app/admin/workers list with status + version columns
/app/admin/workers/new add form with profile dropdown
/app/admin/workers/:id detail with all actions + logs + system update
/app/admin/credentials tabs: AWS credentials + worker profiles,
Releases panel, channel selector +
auto-update toggle in profile form
Warmbly
Email warmup and cold email platform.
Overview
Warmbly is a microservices-based platform for email warmup and cold outreach. It supports Gmail (via API) and IMAP/SMTP providers, with real-time event streaming, distributed workers, and comprehensive tracking.
Architecture
The Frontend (React) connects to three backend services: the Backend API (Go, :8080), Realtime (Elixir/Phoenix, :4000), and Tracking (Rust, :3000).
All three services publish events to Kafka (with Avro/Schema Registry). The Realtime service also uses Google Pub/Sub for cross-node message fanout.
Downstream from Kafka, the Consumer (Go) processes tracking events and the Worker (Go, 1 per machine) handles email operations. Both read/write to PostgreSQL, Redis, and Cassandra (DataStax Astra).
Tech Stack
| Component | Technology |
|---|---|
| Backend API | Go 1.25, Gin |
| Consumer | Go, Kafka consumer |
| Worker | Go, distributed (1 per machine) |
| Tracking | Rust, Axum |
| Realtime | Elixir 1.18, Phoenix Channels |
| Message Queue | Confluent Kafka with Avro/Schema Registry |
| Primary Database | PostgreSQL 16 (AWS RDS) |
| Time-series | DataStax Astra (Cassandra) |
| Cache | Redis 7 |
| Object Storage | AWS S3 |
| Pub/Sub | Google Cloud Pub/Sub |
| Task Queue | Google Cloud Tasks |
Services
| Service | Port | Description |
|---|---|---|
| Backend | 8080 | REST API, authentication, business logic |
| Tracking | 3000 | Pixel tracking and click tracking |
| Realtime | 4000 | WebSocket gateway for real-time events |
| Consumer | - | Kafka event consumer, processes tracking events |
| Worker | - | Distributed worker for email operations |
Quick Start (Development)
Prerequisites
- Docker and Docker Compose
- Go 1.25+ (for local development)
- Rust (for tracking service)
- Elixir 1.18+ (for realtime service)
Start with Docker Compose
cd deploy/docker
# Start infrastructure only (database, redis, kafka)
docker-compose up -d postgres redis kafka schema-registry
# Start all services
docker-compose up
Service URLs (Local)
- Backend API: http://localhost:8080
- Tracking: http://localhost:3000
- Realtime: http://localhost:4000
- Mailpit (email inbox): http://localhost:8025
- Schema Registry: http://localhost:8081
- PostgreSQL: localhost:5432
- Redis: localhost:6379
- Kafka: localhost:9092
Project Structure
cmd/— Service entrypoints:backend/,consumer/,worker/internal/— Core Go code:api/— HTTP handlersapp/— Application services (auth, email, campaign, etc.)config/— Configuration loading (env vars + AWS Secrets Manager)events/— Kafka event schemasinfrastructure/— Database, cache, and queue clientsmodels/— Domain modelsnotify/— Email notification service and templatespkg/— Internal packages (emsg, crypto, etc.)repository/— Data access layer
tracking/— Rust tracking service (Axum)realtime/— Elixir WebSocket service (Phoenix Channels)web/— React frontenddeploy/— Docker Compose, Dockerfiles, Kubernetes manifestsresources/— Technical documentation
Documentation
- Local Development - Docker Compose setup for local development
- Deployment Guide - Step-by-step deployment instructions
- CI/CD Pipeline - CI/CD technical details
- Architecture - System architecture overview
- Events - Kafka event system
- EMSG Format - Email message blob format
- Gmail Integration - Gmail API usage
- IMAP Integration - IMAP/SMTP integration
Building
# Go services (from root)
go build -o bin/backend ./cmd/backend
go build -o bin/consumer ./cmd/consumer
go build -o bin/worker ./cmd/worker
# Tracking service
cd tracking && cargo build --release
# Realtime service
cd realtime && mix deps.get && mix release
CI/CD Pipeline
Warmbly uses GitHub Actions for CI/builds and GitOps for deployments.
PRs trigger CI (tests, linting, security scan). Merging to main builds and pushes Docker images. Tagging a release (v*.*.*) triggers a production deploy via ArgoCD.
| Workflow | Trigger | Purpose |
|---|---|---|
ci.yml |
PR/Push | Tests, linting, security scan |
build-push.yml |
Push to main | Build & push Docker images |
release.yml |
Tag v*.*.* |
Build release images (triggers prod deploy) |
License
Licensed under the Apache License 2.0.
Copyright 2026 Mindroot Ltd
Full license: LICENSE