starred/warmbly
1
0
Fork 0
mirror of https://github.com/warmbly/warmbly.git synced 2026-08-19 00:01:14 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
c9f2ae4a112c09c82e2a5fdd29da170a4df07054
warmbly/resources/cicd.md
T
Matthew Meszaros f4345c3c54 docs: rewrite README and resources for current architecture 
Old docs described a k8s/ArgoCD/Terraform deployment that no longer
exists, with ASCII-art system diagrams that hadn't aged well. Rewritten
to match how the project actually ships:

- README: control plane (Railway) + execution plane (per-VPS workers)
  split, dashboard-driven worker management, credentials/profiles,
  auto-update from GitHub releases, OS package updates, self-hosting
  knobs. Removed all ASCII art.

- resources/architecture.md: control vs execution plane, encryption
  model (worker SSH keys + platform secrets under the same KMS-envelope
  cipher as user secrets), worker identity from public IPv4, credentials
  model, push-driven release flow, anti-abuse layers, source anchors.

- resources/deployment-guide.md: end-to-end from "provision a VPS" to
  "auto-update on release". No more k8s, ArgoCD, kubectl, or Terraform.
  Step-by-step backend env, webhook setup, worker add flow, day-2 ops,
  rollback per plane.

- resources/local-development.md: the five make targets (dev / sim /
  seed / tools / reset), what each profile runs, LocalStack bootstrap,
  rich seed contents, native dev against containerized infra, the
  offset-port URL table.

- resources/cicd.md: the two-plane build/release flow, image tag scheme
  ({sha} / dev / vX.Y.Z / vX.Y / vX / prod), webhook setup, release
  process, security notes around HMAC and least-privilege worker AWS
  keys.

- deploy/README.md: tight version of the same.
2026-05-18 13:09:31 +00:00

183 lines
7.7 KiB
Markdown
Raw Blame History

# CI / Release Flow
Two distinct flows for the two planes.
| Plane | Build | Deploy |
|-------|-------|--------|
| Control plane (backend, consumer, tracking, realtime, web) | GitHub Actions builds + pushes images to GHCR | Railway auto-deploys on push to main |
| Workers | GitHub Actions builds + pushes images to GHCR | Admin dashboard rolls each VPS over SSH; optional auto-update via GitHub release webhook |
The two flows are deliberately separate. Workers can't be updated by re-deploying the control plane because they live on independent VPSes.
## Workflows
| File | Trigger | What it does |
|------|---------|--------------|
| `.github/workflows/ci.yml` | PR + push | Tests, linting, security scan |
| `.github/workflows/build-push.yml` | Push to `main` | Builds changed services, pushes `:{sha}` and `:dev` tags to GHCR |
| `.github/workflows/release.yml` | Tag `v*.*.*` | Builds all services with full semver tags + creates GitHub Release |
Image registry: `ghcr.io/<owner>/warmbly/{backend,consumer,worker,tracking,realtime}`.
## CI
`ci.yml` runs on PRs and pushes to main:
- **Change detection** via `dorny/paths-filter` so unchanged services are skipped
- **Go**: `golangci-lint run` + `go test ./...`
- **Rust** (tracking): `cargo fmt --check` + `cargo clippy -- -D warnings` + `cargo test`
- **Elixir** (realtime): `mix format --check-formatted` + `mix credo --strict` + `mix test`
- **Security**: Trivy scan against built images
Required to pass for merge.
## Build & push on main
`build-push.yml` runs on push to `main`:
1. Detect which service directories changed (`go.mod`, `internal/`, `cmd/<svc>/`, `deploy/docker/<svc>.Dockerfile`, etc.)
2. For each changed service, build the Docker image with buildx
3. Push to GHCR with two tags:
- `:{sha}` — exact commit
- `:dev` — moving pointer to the latest main build
Multi-arch (`linux/amd64,linux/arm64`) with `cache-from`/`cache-to: type=gha`.
## Release
`release.yml` runs on tag push matching `v*.*.*`:
1. Extract version from the tag (`v1.2.3` → `1.2.3`)
2. Build all five services
3. Push with four tags each:
- `:vX.Y.Z` — exact
- `:vX.Y` — minor pointer
- `:vX` — major pointer
- `:prod` — moving pointer to the latest release
4. Create a GitHub Release with autogenerated notes and an image table
The `:prod` and `:dev` tags are what the auto-update flow consumes.
## Deploying the control plane
Production runs on **Railway**. Each control-plane service is a Railway service in one project. Pushes to `main` trigger Railway's GitHub integration, which pulls the `:dev` image (or whatever you've configured the service to track) and rolls.
To pin production to release tags instead of `:dev`, set the Railway service's image to `ghcr.io/<owner>/warmbly/<service>:prod` (auto-updates on every release) or `:vX.Y.Z` (manual bumps).
Self-hosters running elsewhere (ECS / Fly / VPS) — the Dockerfiles in `deploy/docker/` are the same; point your hosting platform at the image and the env vars in `deploy/config/env.example`.
## Deploying workers
Workers don't get touched by Railway. They live on VPSes and update through one of three paths.
### 1. Manual install on the VPS
```bash
curl -fsSL https://get.example.com/worker | sudo bash -s -- \
--kafka ... --schema-registry ... --redis ... \
--aws-region ... --aws-key ... --aws-secret ...
```
Run once when adding a new VPS to the fleet. Identity is derived from the VPS's public IPv4.
### 2. Admin dashboard
Add Worker → paste host/port/user → backend generates a keypair → paste public key into VPS → click Test → Install. From there, all subsequent operations are dashboard buttons:
- Restart, Update image, Apply config, Uninstall, Rotate SSH keys, System updates, Reboot, Tail logs
### 3. Auto-update on GitHub release
Each worker profile picks a release channel:
- `pinned` — admin sets the image tag manually
- `stable` — backend tracks the latest non-prerelease release
- `dev` — backend tracks the latest release (including prereleases)
Trigger model is push-driven:
- **GitHub webhook**: `POST /webhooks/github/releases` on every release event, HMAC-validated with `RELEASES_WEBHOOK_SECRET`.
- **Backend boot**: a single check at startup so the dashboard isn't empty after a redeploy.
- **Manual**: admin "Check now" button.
When a new release fires the webhook, the backend resolves each profile's channel to a concrete image. If the profile has `auto_update=true`, each assigned worker is rolled immediately: the orchestrator SSHes in, runs `install-worker.sh --update --image <new>` (which regenerates the systemd unit, pulls the image, restarts the container), and records the version. If `auto_update=false`, the dashboard surfaces an "update available" badge and waits for a click.
All release config is env-driven so self-hosters can point at their own fork:
```
RELEASES_ENABLED=true
RELEASES_GITHUB_REPO=youruser/yourfork
RELEASES_WORKER_IMAGE_REPO=ghcr.io/youruser/yourfork/worker
RELEASES_WEBHOOK_SECRET=<shared secret>
RELEASES_GITHUB_TOKEN=<optional PAT>
```
## Setting up the release webhook
1. GitHub repo → Settings → Webhooks → Add webhook
2. Payload URL: `https://api.example.com/webhooks/github/releases`
3. Content type: `application/json`
4. Secret: same value as `RELEASES_WEBHOOK_SECRET`
5. Events: select **Releases** only
6. Save
Test by republishing an existing release — backend should log `releases: ...` and the dashboard's Releases panel updates.
## Image tag scheme
| Tag | Source | Use |
|-----|--------|-----|
| `:{sha}` | push to main | Pin a specific build for debugging |
| `:dev` | latest main | Default for staging / preview |
| `:vX.Y.Z` | tag push | Exact release |
| `:vX.Y` | tag push | Minor-track auto-update |
| `:vX` | tag push | Major-track auto-update |
| `:prod` | latest release | Default for production |
The auto-update flow always resolves to a concrete `:vX.Y.Z`, never a moving tag, so the recorded `workers.image_version` is meaningful.
## Releasing
```bash
git tag -a v1.2.3 -m "Release v1.2.3"
git push origin v1.2.3
```
What happens:
1. `release.yml` builds and pushes images
2. GitHub creates a Release
3. GitHub webhook fires
4. Backend resolves channels:
- `stable` profiles → `:v1.2.3`
- `dev` profiles → `:v1.2.3` if it's not a prerelease, otherwise unchanged
5. Profiles with `auto_update=true` start rolling assigned workers
6. Dashboard shows updated state in real time
## Rollback
**Control plane**: Railway has a one-click rollback in its UI to a previous deployment.
**Workers**: edit the profile, set channel to `pinned`, set `worker_image` to a specific older tag (e.g. `ghcr.io/.../worker:v1.2.2`), save, click Apply. All assigned workers roll back over SSH.
## Troubleshooting
```bash
gh run list --workflow=build-push.yml
gh run view <run-id> --log
gh run list --workflow=release.yml
```
Webhook not firing:
- GitHub repo → Settings → Webhooks → Recent Deliveries shows every dispatch and its response
- A 401 means the secret is wrong
- A 200 with no resulting rollout means the backend received it but couldn't reach GitHub or couldn't decrypt one of the affected profile's secrets; check `journalctl -u backend` or your Railway logs
## Security notes
- Worker SSH private keys are encrypted at rest via KMS-wrapped DEK (same envelope as user secrets).
- The release webhook authenticates via HMAC-SHA256, not bearer token, so the secret never appears in logs.
- All worker runtime credentials (Kafka SASL, Schema Registry secret, Redis URL, AWS secret access key) are stored encrypted; the dashboard only ever sees "set / not set" booleans for sensitive fields.
- Worker AWS keys should be least-privilege: KMS Decrypt, S3 read/write to the configured bucket, DynamoDB read/write to `UserEncryptedKeys` + `EmailMessageData`, plus Kafka SASL. Nothing else.
Reference in New Issue View Git Blame Copy Permalink