warmbly/resources/cicd.md

# CI / Release Flow

Two distinct flows for the two planes.

| Plane | Build | Deploy |
|-------|-------|--------|
| Control plane (backend, consumer, tracking, realtime, web) | GitHub Actions builds + pushes images to GHCR | Railway auto-deploys on push to main |
| Workers | GitHub Actions builds + pushes images to GHCR | Admin dashboard rolls each VPS over SSH; optional auto-update via GitHub release webhook |

The two flows are deliberately separate. Workers can't be updated by re-deploying the control plane because they live on independent VPSes.

## Workflows

| File | Trigger | What it does |
|------|---------|--------------|
| `.github/workflows/ci.yml` | PR + push | Tests, linting, security scan |
| `.github/workflows/build-push.yml` | Push to `main` | Builds changed services, pushes `:{sha}` and `:dev` tags to GHCR |
| `.github/workflows/release.yml` | Tag `v*.*.*` | Builds all services with full semver tags + creates GitHub Release |

Image registry: `ghcr.io/<owner>/warmbly/{backend,consumer,worker,tracking,realtime}`.

## CI

`ci.yml` runs on PRs and pushes to main:

- **Change detection** via `dorny/paths-filter` so unchanged services are skipped
- **Go**: `golangci-lint run` + `go test ./...`
- **Rust** (tracking): `cargo fmt --check` + `cargo clippy -- -D warnings` + `cargo test`
- **Elixir** (realtime): `mix format --check-formatted` + `mix credo --strict` + `mix test`
- **Security**: Trivy scan against built images

Required to pass for merge.

## Build & push on main

`build-push.yml` runs on push to `main`:

1. Detect which service directories changed (`go.mod`, `internal/`, `cmd/<svc>/`, `deploy/docker/<svc>.Dockerfile`, etc.)
2. For each changed service, build the Docker image with buildx
3. Push to GHCR with two tags:
   - `:{sha}` — exact commit
   - `:dev` — moving pointer to the latest main build

Multi-arch (`linux/amd64,linux/arm64`) with `cache-from`/`cache-to: type=gha`.

## Release

`release.yml` runs on tag push matching `v*.*.*`:

1. Extract version from the tag (`v1.2.3` → `1.2.3`)
2. Build all five services
3. Push with four tags each:
   - `:vX.Y.Z` — exact
   - `:vX.Y` — minor pointer
   - `:vX` — major pointer
   - `:prod` — moving pointer to the latest release
4. Create a GitHub Release with autogenerated notes and an image table

The `:prod` and `:dev` tags are what the auto-update flow consumes.

## Deploying the control plane

Production runs on **Railway**. Each control-plane service is a Railway service in one project. Pushes to `main` trigger Railway's GitHub integration, which pulls the `:dev` image (or whatever you've configured the service to track) and rolls.

To pin production to release tags instead of `:dev`, set the Railway service's image to `ghcr.io/<owner>/warmbly/<service>:prod` (auto-updates on every release) or `:vX.Y.Z` (manual bumps).

Self-hosters running elsewhere (ECS / Fly / VPS) — the Dockerfiles in `deploy/docker/` are the same; point your hosting platform at the image and the env vars in `deploy/config/env.example`.

## Deploying workers

Workers don't get touched by Railway. They live on VPSes and update through one of three paths.

### 1. Manual install on the VPS

```bash
curl -fsSL https://get.example.com/worker | sudo bash -s -- \
  --kafka ... --schema-registry ... --redis ... \
  --aws-region ... --aws-key ... --aws-secret ...
```

Run once when adding a new VPS to the fleet. Identity is derived from the VPS's public IPv4.

### 2. Admin dashboard

Add Worker → paste host/port/user → backend generates a keypair → paste public key into VPS → click Test → Install. From there, all subsequent operations are dashboard buttons:

- Restart, Update image, Apply config, Uninstall, Rotate SSH keys, System updates, Reboot, Tail logs

### 3. Auto-update on GitHub release

Each worker profile picks a release channel:

- `pinned` — admin sets the image tag manually
- `stable` — backend tracks the latest non-prerelease release
- `dev` — backend tracks the latest release (including prereleases)

Trigger model is push-driven:

- **GitHub webhook**: `POST /webhooks/github/releases` on every release event, HMAC-validated with `RELEASES_WEBHOOK_SECRET`.
- **Backend boot**: a single check at startup so the dashboard isn't empty after a redeploy.
- **Manual**: admin "Check now" button.

When a new release fires the webhook, the backend resolves each profile's channel to a concrete image. If the profile has `auto_update=true`, each assigned worker is rolled immediately: the orchestrator SSHes in, runs `install-worker.sh --update --image <new>` (which regenerates the systemd unit, pulls the image, restarts the container), and records the version. If `auto_update=false`, the dashboard surfaces an "update available" badge and waits for a click.

All release config is env-driven so self-hosters can point at their own fork:

```
RELEASES_ENABLED=true
RELEASES_GITHUB_REPO=youruser/yourfork
RELEASES_WORKER_IMAGE_REPO=ghcr.io/youruser/yourfork/worker
RELEASES_WEBHOOK_SECRET=<shared secret>
RELEASES_GITHUB_TOKEN=<optional PAT>
```

## Setting up the release webhook

1. GitHub repo → Settings → Webhooks → Add webhook
2. Payload URL: `https://api.example.com/webhooks/github/releases`
3. Content type: `application/json`
4. Secret: same value as `RELEASES_WEBHOOK_SECRET`
5. Events: select **Releases** only
6. Save

Test by republishing an existing release — backend should log `releases: ...` and the dashboard's Releases panel updates.

## Image tag scheme

| Tag | Source | Use |
|-----|--------|-----|
| `:{sha}` | push to main | Pin a specific build for debugging |
| `:dev` | latest main | Default for staging / preview |
| `:vX.Y.Z` | tag push | Exact release |
| `:vX.Y` | tag push | Minor-track auto-update |
| `:vX` | tag push | Major-track auto-update |
| `:prod` | latest release | Default for production |

The auto-update flow always resolves to a concrete `:vX.Y.Z`, never a moving tag, so the recorded `workers.image_version` is meaningful.

## Releasing

```bash
git tag -a v1.2.3 -m "Release v1.2.3"
git push origin v1.2.3
```

What happens:

1. `release.yml` builds and pushes images
2. GitHub creates a Release
3. GitHub webhook fires
4. Backend resolves channels:
   - `stable` profiles → `:v1.2.3`
   - `dev` profiles → `:v1.2.3` if it's not a prerelease, otherwise unchanged
5. Profiles with `auto_update=true` start rolling assigned workers
6. Dashboard shows updated state in real time

## Rollback

**Control plane**: Railway has a one-click rollback in its UI to a previous deployment.

**Workers**: edit the profile, set channel to `pinned`, set `worker_image` to a specific older tag (e.g. `ghcr.io/.../worker:v1.2.2`), save, click Apply. All assigned workers roll back over SSH.

## Troubleshooting

```bash
gh run list --workflow=build-push.yml
gh run view <run-id> --log
gh run list --workflow=release.yml
```

Webhook not firing:

- GitHub repo → Settings → Webhooks → Recent Deliveries shows every dispatch and its response
- A 401 means the secret is wrong
- A 200 with no resulting rollout means the backend received it but couldn't reach GitHub or couldn't decrypt one of the affected profile's secrets; check `journalctl -u backend` or your Railway logs

## Security notes

- Worker SSH private keys are encrypted at rest via KMS-wrapped DEK (same envelope as user secrets).
- The release webhook authenticates via HMAC-SHA256, not bearer token, so the secret never appears in logs.
- All worker runtime credentials (Kafka SASL, Schema Registry secret, Redis URL, AWS secret access key) are stored encrypted; the dashboard only ever sees "set / not set" booleans for sensitive fields.
- Worker AWS keys should be least-privilege: KMS Decrypt, S3 read/write to the configured bucket, plus Kafka SASL. Nothing else. (Workers no longer use DynamoDB — encrypted DEKs and the message-ID map are reached over the backend's internal API.)