warmbly/resources/cicd.md

CI / Release Flow

Two distinct flows for the two planes.

Plane	Build	Deploy
Control plane (backend, consumer, tracking, realtime, web)	GitHub Actions builds + pushes images to GHCR	Railway auto-deploys on push to main
Workers	GitHub Actions builds + pushes images to GHCR	Admin dashboard rolls each VPS over SSH; optional auto-update via GitHub release webhook

The two flows are deliberately separate. Workers can't be updated by re-deploying the control plane because they live on independent VPSes.

Workflows

File	Trigger	What it does
.github/workflows/ci.yml	PR + push	Tests, linting, security scan
.github/workflows/build-push.yml	Push to main	Builds changed services, pushes :{sha} and :dev tags to GHCR
.github/workflows/release.yml	Tag v*.*.*	Builds all services with full semver tags + creates GitHub Release

Image registry: ghcr.io/<owner>/warmbly/{backend,consumer,worker,tracking,realtime}.

CI

ci.yml runs on PRs and pushes to main:

• Change detection via dorny/paths-filter so unchanged services are skipped
• Go: golangci-lint run + go test ./...
• Rust (tracking): cargo fmt --check + cargo clippy -- -D warnings + cargo test
• Elixir (realtime): mix format --check-formatted + mix credo --strict + mix test
• Security: Trivy scan against built images

Required to pass for merge.

Build & push on main

build-push.yml runs on push to main:

1. Detect which service directories changed (go.mod, internal/, cmd/<svc>/, deploy/docker/<svc>.Dockerfile, etc.)
2. For each changed service, build the Docker image with buildx
3. Push to GHCR with two tags:
   • :{sha} — exact commit
   • :dev — moving pointer to the latest main build

Multi-arch (linux/amd64,linux/arm64) with cache-from/cache-to: type=gha.

Release

release.yml runs on tag push matching v*.*.*:

1. Extract version from the tag (v1.2.3 → 1.2.3)
2. Build all five services
3. Push with four tags each:
   • :vX.Y.Z — exact
   • :vX.Y — minor pointer
   • :vX — major pointer
   • :prod — moving pointer to the latest release
4. Create a GitHub Release with autogenerated notes and an image table

The :prod and :dev tags are what the auto-update flow consumes.

Deploying the control plane

Production runs on Railway. Each control-plane service is a Railway service in one project. Pushes to main trigger Railway's GitHub integration, which pulls the :dev image (or whatever you've configured the service to track) and rolls.

To pin production to release tags instead of :dev, set the Railway service's image to ghcr.io/<owner>/warmbly/<service>:prod (auto-updates on every release) or :vX.Y.Z (manual bumps).

Self-hosters running elsewhere (ECS / Fly / VPS) — the Dockerfiles in deploy/docker/ are the same; point your hosting platform at the image and the env vars in deploy/config/env.example.

Deploying workers

Workers don't get touched by Railway. They live on VPSes and update through one of three paths.

1. Manual install on the VPS

curl -fsSL https://get.example.com/worker | sudo bash -s -- \
  --kafka ... --schema-registry ... --redis ... \
  --aws-region ... --aws-key ... --aws-secret ...

Run once when adding a new VPS to the fleet. Identity is derived from the VPS's public IPv4.

2. Admin dashboard

Add Worker → paste host/port/user → backend generates a keypair → paste public key into VPS → click Test → Install. From there, all subsequent operations are dashboard buttons:

• Restart, Update image, Apply config, Uninstall, Rotate SSH keys, System updates, Reboot, Tail logs

3. Auto-update on GitHub release

Each worker profile picks a release channel:

• pinned — admin sets the image tag manually
• stable — backend tracks the latest non-prerelease release
• dev — backend tracks the latest release (including prereleases)

Trigger model is push-driven:

• GitHub webhook: POST /webhooks/github/releases on every release event, HMAC-validated with RELEASES_WEBHOOK_SECRET.
• Backend boot: a single check at startup so the dashboard isn't empty after a redeploy.
• Manual: admin "Check now" button.

When a new release fires the webhook, the backend resolves each profile's channel to a concrete image. If the profile has auto_update=true, each assigned worker is rolled immediately: the orchestrator SSHes in, runs install-worker.sh --update --image <new> (which regenerates the systemd unit, pulls the image, restarts the container), and records the version. If auto_update=false, the dashboard surfaces an "update available" badge and waits for a click.

All release config is env-driven so self-hosters can point at their own fork:

RELEASES_ENABLED=true
RELEASES_GITHUB_REPO=youruser/yourfork
RELEASES_WORKER_IMAGE_REPO=ghcr.io/youruser/yourfork/worker
RELEASES_WEBHOOK_SECRET=<shared secret>
RELEASES_GITHUB_TOKEN=<optional PAT>

Setting up the release webhook

1. GitHub repo → Settings → Webhooks → Add webhook
2. Payload URL: https://api.example.com/webhooks/github/releases
3. Content type: application/json
4. Secret: same value as RELEASES_WEBHOOK_SECRET
5. Events: select Releases only
6. Save

Test by republishing an existing release — backend should log releases: ... and the dashboard's Releases panel updates.

Image tag scheme

Tag	Source	Use
:{sha}	push to main	Pin a specific build for debugging
:dev	latest main	Default for staging / preview
:vX.Y.Z	tag push	Exact release
:vX.Y	tag push	Minor-track auto-update
:vX	tag push	Major-track auto-update
:prod	latest release	Default for production

The auto-update flow always resolves to a concrete :vX.Y.Z, never a moving tag, so the recorded workers.image_version is meaningful.

Releasing

git tag -a v1.2.3 -m "Release v1.2.3"
git push origin v1.2.3

What happens:

1. release.yml builds and pushes images
2. GitHub creates a Release
3. GitHub webhook fires
4. Backend resolves channels:
   • stable profiles → :v1.2.3
   • dev profiles → :v1.2.3 if it's not a prerelease, otherwise unchanged
5. Profiles with auto_update=true start rolling assigned workers
6. Dashboard shows updated state in real time

Rollback

Control plane: Railway has a one-click rollback in its UI to a previous deployment.

Workers: edit the profile, set channel to pinned, set worker_image to a specific older tag (e.g. ghcr.io/.../worker:v1.2.2), save, click Apply. All assigned workers roll back over SSH.

Troubleshooting

gh run list --workflow=build-push.yml
gh run view <run-id> --log
gh run list --workflow=release.yml

Webhook not firing:

• GitHub repo → Settings → Webhooks → Recent Deliveries shows every dispatch and its response
• A 401 means the secret is wrong
• A 200 with no resulting rollout means the backend received it but couldn't reach GitHub or couldn't decrypt one of the affected profile's secrets; check journalctl -u backend or your Railway logs

Security notes

• Worker SSH private keys are encrypted at rest via KMS-wrapped DEK (same envelope as user secrets).
• The release webhook authenticates via HMAC-SHA256, not bearer token, so the secret never appears in logs.
• All worker runtime credentials (Kafka SASL, Schema Registry secret, Redis URL, AWS secret access key) are stored encrypted; the dashboard only ever sees "set / not set" booleans for sensitive fields.
• Worker AWS keys should be least-privilege: KMS Decrypt, S3 read/write to the configured bucket, DynamoDB read/write to UserEncryptedKeys + EmailMessageData, plus Kafka SASL. Nothing else.