diff --git a/AGENTS.md b/AGENTS.md index dac2f829..ef404fad 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -285,7 +285,7 @@ API keys with the `REALTIME_SUBSCRIBE` permission (bit 11) can connect to the sa - `web/`: in-product frontend (dashboard). Customer-facing only: it holds no platform-admin screens, and operator tooling must not be added back here - `admin/`: platform admin panel (:5174), the single operator surface. Workers, users, orgs, warmup, campaigns, analytics, audit. Every route sits behind `RequireAdmin` and the backend's `RequireAdminPermission` gates - `site/`: public marketing site (Astro 5 + Tailwind v4). `site/public/install.sh` is the self-host installer served at warmbly.com/install.sh and `site/public/cli.sh` is the CLI installer served at warmbly.com/cli.sh (with `cli.ps1` for Windows), each with its checksum next to it; see the rules above before touching either -- `deploy/`: production deploy manifests, infrastructure, and runtime config +- `deploy/`: production deploy manifests, infrastructure, and runtime config. `deploy/split-cloud/` is the three-provider shape (control plane on a container host, bus + cache + fleet on machines you own, database + root key + object store in a cloud region), documented at `docs/content/docs/development/split-deployment.mdx` - `docs/`: documentation site (docs.warmbly.com); product guides, API reference, and self-hosting/engineering docs under `content/docs/development/` - `scripts/`: one-off tooling (codegen, migrations, installer checks, local dev utilities) - `skills/`: agent playbooks shipped with the repo (`warmbly-cli` for the `warmbly` CLI, `warmbly-api` for the same product surface through `warmblyctl`, `warmbly-ops` for instance administration, `warmbly-install` for standing an instance up and moving it). A command an operator can run is not usable by an agent until it is in one of these @@ -368,7 +368,13 @@ Two rules that follow from those: - **systemd runs no shell.** No `$(...)`, no globbing, no word splitting in a unit. A value that has to vary comes from an `EnvironmentFile` as `${VAR}`, which expands to exactly one argument - **What the node may write and what root reads are different directories.** The container runs as uid 1000; it gets `/var/lib/warmbly/node` and nothing else. `image-ref` lives one level up, root-owned, because systemd feeds it to a root `docker run --network host` and a node that could rewrite it would choose the image root executes -The env the join endpoint hands a node is rendered from the backend's own environment (`nodeEnvKeys` in `internal/api/handler/fleet_nodes.go`). `PRIMARY_DB` is deliberately absent: a worker reaches relational data through the internal API and nothing else, and shipping a DSN here would quietly undo that boundary. +The env the join endpoint hands a node is rendered from the backend's own environment (`nodeEnvKeys` in `internal/api/handler/fleet_nodes.go`). Three things are decided rather than copied: + +- **`PRIMARY_DB` reaches a consumer and never a worker.** A worker gets relational data through the internal API and nothing else; a consumer opens Postgres itself and cannot boot without the DSN. Role is known at render time, so the exclusion lives exactly where it belongs +- **The crypto and blob providers are translated, not copied** (`nodeProviders`), so no machine in the fleet carries a cloud credential +- **Every name sent must be one the node's own code reads.** `S3_BUCKET` and `KMS_KEY_ID` were sent for a while and read by nothing, against a storage layer reading `BLOB_BUCKET` and a KMS factory reading `KMS_AWS_KEY_ID`, so an AWS-backed node silently used the default bucket and the default key alias. `internal/api/handler/fleet_nodes_test.go` asserts on the rendered file + +`/etc/warmbly/node.local.env` is the operator's half: created once by `join.sh`, never rewritten, and passed to the container after `node.env` so it wins. That is where a value the control plane cannot know belongs, and it is why nothing needs to be hand-edited into a file the next join replaces. Operator surface: `warmblyctl fleet` (join-token, list, show, remove, pin, version, channel) and the admin panel's Fleet section. There is no install, restart, logs or reboot action anywhere, because nothing reaches into a machine. @@ -399,6 +405,7 @@ Design intent: - workers may talk to infrastructure-style services that scale independently, such as S3, KMS, and cache layers - relational data the worker needs (encrypted DEKs, the messageId→internal-email map) is reached over the backend's internal HTTP API (`/api/v1/internal/...`), never via direct SQL - worker-local state should be minimal and disposable +- **a node holds no cloud credential.** The two privileged operations it needs are brokered through the internal API: `KMS_PROVIDER=brokered` posts sealed keys to `/api/v1/internal/dek/decrypt` and `BLOB_PROVIDER=brokered` asks `/api/v1/internal/blobs/presign` to sign one operation on one key. `renderNodeEnv` translates `aws`/`s3` into these automatically when rendering a node's env, so an IAM key never reaches a machine in the fleet. Blob bytes still travel node↔store directly; only the signature comes from the control plane Current code matches that intent in `cmd/worker/main.go`: the worker boots Kafka, Redis cache, KMS, and S3 clients, and reaches DEKs + the email message map through the backend's internal API, but does not open a PostgreSQL connection. @@ -432,7 +439,10 @@ Main code paths: - `internal/infrastructure/kms/encryption.go` - `internal/infrastructure/kms/decryption.go` - `internal/infrastructure/encryptedkeys/` (`store.go`, `factory.go`, `postgres.go`, `http.go`) -- `internal/api/handler/internal_dek.go` (the worker-facing DEK proxy endpoint) +- `internal/infrastructure/kms/brokered.go` (the node-side provider that holds no key material) +- `internal/infrastructure/storage/brokered.go` (the node-side blob store that holds no bucket credential) +- `internal/api/handler/internal_dek.go` (the worker-facing DEK proxy endpoint, and the decrypt broker) +- `internal/api/handler/internal_blobs.go` (the blob presign broker) Operational guidance: diff --git a/cmd/backend/main.go b/cmd/backend/main.go index e74817dc..d74a9529 100644 --- a/cmd/backend/main.go +++ b/cmd/backend/main.go @@ -287,6 +287,9 @@ func main() { // repository / object-storage needs. Declared up here so they // survive the config block where they're initialized. var s3ForHandler storage.Store + // The root of trust, surfaced so /api/v1/internal/dek/decrypt can open a + // sealed key for a node that carries no KMS credential of its own. + var kmsForHandler kms.Provider var emailMessageMapForHandler repository.EmailMessageMapRepository var emailSyncStateRepository repository.EmailSyncStateRepository var trackedLinkRepository repository.TrackedLinkRepository @@ -379,6 +382,7 @@ func main() { errs.CaptureFatal(err) log.Fatal(err) } + kmsForHandler = kms geoPath, err := cfg.LoadGeoDBPath(ctx) if err != nil { @@ -1982,6 +1986,7 @@ func main() { // without a dedicated service layer (avatars, etc.). Storage: s3ForHandler, EncryptedKeys: encryptedKeys, + KMS: kmsForHandler, EmailMessageMap: emailMessageMapForHandler, EmailSyncState: emailSyncStateRepository, TrackedLinks: trackedLinkRepository, diff --git a/deploy/README.md b/deploy/README.md index e61c04e5..351388e3 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -60,6 +60,11 @@ Full reference: [local development](https://docs.warmbly.com/development/local-d ## Deploying without Docker +`deploy/split-cloud/` holds the manifests for running the two planes on +different providers: the control plane on a container host, the bus and cache +and fleet on machines you own, and the database, root key and object store in a +cloud region. Its README lists what is in it. + `deploy/systemd/` holds one unit per service and `deploy/nginx/warmbly.conf` a site that serves the static frontends and proxies the API, websocket and tracking hosts. The step-by-step guide that uses them is [Deploying without Docker](https://docs.warmbly.com/development/bare-metal/). ## Deploying the control plane diff --git a/deploy/split-cloud/README.md b/deploy/split-cloud/README.md new file mode 100644 index 00000000..ea5f8795 --- /dev/null +++ b/deploy/split-cloud/README.md @@ -0,0 +1,52 @@ +# Split deployment + +The control plane on a container host, the fleet on machines you own, and the +database, root key and object store in a cloud region next to the control +plane. Three providers, one instance. + +``` +container host your machines cloud region +────────────── ───────────── ──────────── +backend worker Postgres +consumer worker KMS key +realtime nats + redis S3 bucket +tracking SES +forms +web / admin +``` + +The full walkthrough, with costs and the order to build it in, is +[Split deployment](https://docs.warmbly.com/development/split-deployment/). + +## What is here + +| Path | What it is | +|------|------------| +| `control-plane.env.example` | Every setting the container host needs, annotated | +| `bus/docker-compose.yml` | NATS JetStream and Redis, both over TLS | +| `bus/nats.conf` | The bus config the compose file mounts | +| `bus/certbot-deploy-hook.sh` | Publishes renewed certificates where the containers can read them | +| `node/docker-compose.yml` | A worker run by hand, for the version-controlled path | +| `node/worker.env.example` | What a worker needs, and what it deliberately does not | +| `../../scripts/aws-bootstrap.sh` | The KMS key, bucket, database and IAM policies | + +## Two things that are not optional + +**Redis over TLS.** The cache holds each organization's decrypted data key for +the life of its entry. A plaintext connection across the internet publishes key +material, and a password does not change that. + +**S3 rather than filesystem blobs.** A worker reads the message body the +backend wrote. On another machine it has neither the disk nor the permissions, +so sends fail at the last step with everything else looking healthy. + +## What a node does not get + +A joining node is handed `KMS_PROVIDER=brokered` and `BLOB_PROVIDER=brokered` +rather than the control plane's own AWS providers, so no machine in the fleet +carries a cloud credential. It opens sealed keys and signs blob operations +through the internal API, with the instance-scoped token it already holds. +Blob bytes still travel directly between the node and the object store. + +A worker also never receives `PRIMARY_DB`. A consumer does, because it is +control plane and updates relational state itself. diff --git a/deploy/split-cloud/bus/certbot-deploy-hook.sh b/deploy/split-cloud/bus/certbot-deploy-hook.sh new file mode 100644 index 00000000..ace1a4cd --- /dev/null +++ b/deploy/split-cloud/bus/certbot-deploy-hook.sh @@ -0,0 +1,30 @@ +#!/bin/sh +# Certbot deploy hook: publish a renewed certificate where the containers can +# read it, then restart the two services that hold it open. +# +# Install as /etc/letsencrypt/renewal-hooks/deploy/warmbly-bus.sh, mode 0755. +# +# Certbot's own live directory is root-owned and 0700 on the private key, which +# neither container can read: NATS runs as uid 1000 and Redis as uid 999. The +# copy exists to widen that deliberately and in one place, rather than by +# loosening /etc/letsencrypt. +set -eu + +DOMAIN="${WARMBLY_BUS_DOMAIN:-bus.example.com}" +SRC="/etc/letsencrypt/live/$DOMAIN" +DEST="/opt/warmbly/certs" + +[ -d "$SRC" ] || { echo "no certificate at $SRC" >&2; exit 1; } + +mkdir -p "$DEST" +cp "$SRC/fullchain.pem" "$DEST/fullchain.pem" +cp "$SRC/chain.pem" "$DEST/chain.pem" +cp "$SRC/privkey.pem" "$DEST/privkey.pem" + +# World-readable on a box whose only job is this. Narrow it to a shared group +# if anything else ever runs here. +chmod 0644 "$DEST/fullchain.pem" "$DEST/chain.pem" +chmod 0644 "$DEST/privkey.pem" + +# Both hold the certificate open and neither re-reads it on its own. +cd /opt/warmbly/bus && docker compose restart nats redis diff --git a/deploy/split-cloud/bus/docker-compose.yml b/deploy/split-cloud/bus/docker-compose.yml new file mode 100644 index 00000000..9de01984 --- /dev/null +++ b/deploy/split-cloud/bus/docker-compose.yml @@ -0,0 +1,70 @@ +# The bus box: NATS JetStream and Redis, reachable by the control plane on one +# host and the fleet on others. +# +# Run this on a machine of its own once you have more than one worker. The +# whole control plane depends on it, so it should not also be the box you +# restart when you redeploy a worker. +# +# Before the first start: +# 1. point bus.example.com at this machine +# 2. get a certificate for it (certbot certonly --standalone -d bus.example.com) +# 3. install certbot-deploy-hook.sh so renewals land in /opt/warmbly/certs +# 4. write .env next to this file with NATS_TOKEN and REDIS_PASSWORD +# +# NATS_TOKEN=$(openssl rand -hex 32) +# REDIS_PASSWORD=$(openssl rand -hex 32) + +services: + nats: + image: nats:2.10-alpine + restart: unless-stopped + command: ["-c", "/etc/nats/nats.conf", "-js", "-sd", "/data", "-m", "8222"] + environment: + NATS_TOKEN: ${NATS_TOKEN:?set NATS_TOKEN in .env} + volumes: + - ./nats.conf:/etc/nats/nats.conf:ro + - /opt/warmbly/certs:/certs:ro + - nats_data:/data + ports: + - "4222:4222" + healthcheck: + test: ["CMD", "wget", "--spider", "-q", "http://localhost:8222/healthz"] + interval: 10s + timeout: 3s + retries: 5 + + redis: + image: redis:7-alpine + restart: unless-stopped + # TLS is not optional on this box. Redis holds each organization's + # decrypted data key for the life of its cache entry, so a plaintext + # connection across the internet publishes key material. The password + # protects access; only TLS protects the traffic. + # + # 6379 stays open for containers on this host and is never published; + # 6380 is the port the control plane connects to. + command: > + redis-server + --port 6379 + --tls-port 6380 + --tls-cert-file /certs/fullchain.pem + --tls-key-file /certs/privkey.pem + --tls-ca-cert-file /certs/chain.pem + --tls-auth-clients no + --requirepass ${REDIS_PASSWORD:?set REDIS_PASSWORD in .env} + --appendonly yes + --maxmemory-policy noeviction + volumes: + - /opt/warmbly/certs:/certs:ro + - redis_data:/data + ports: + - "6380:6380" + healthcheck: + test: ["CMD", "redis-cli", "-a", "${REDIS_PASSWORD}", "ping"] + interval: 10s + timeout: 3s + retries: 5 + +volumes: + nats_data: + redis_data: diff --git a/deploy/split-cloud/bus/nats.conf b/deploy/split-cloud/bus/nats.conf new file mode 100644 index 00000000..9d5f932e --- /dev/null +++ b/deploy/split-cloud/bus/nats.conf @@ -0,0 +1,31 @@ +# NATS JetStream for a split deployment: the control plane runs on a container +# host, the fleet runs on machines you own, and both reach this. +# +# The bus crosses the public internet here, so it is TLS and it is +# authenticated. A token in the URL is what every Warmbly service (including +# the Rust tracking publisher) already understands: +# +# NATS_URL=tls://@bus.example.com:4222 + +listen: 0.0.0.0:4222 + +# Monitoring stays on loopback. It exposes stream and consumer state to anyone +# who can reach it, and nothing outside this machine needs it. +http: 127.0.0.1:8222 + +jetstream { + store_dir: "/data" + # Give JetStream a bound rather than the whole disk, so a stuck consumer + # fills a quota instead of the filesystem the container runtime is on. + max_file_store: 8GB +} + +authorization { + # openssl rand -hex 32 + token: $NATS_TOKEN +} + +tls { + cert_file: "/certs/fullchain.pem" + key_file: "/certs/privkey.pem" +} diff --git a/deploy/split-cloud/control-plane.env.example b/deploy/split-cloud/control-plane.env.example new file mode 100644 index 00000000..8384d65e --- /dev/null +++ b/deploy/split-cloud/control-plane.env.example @@ -0,0 +1,78 @@ +# The control plane: backend, consumer, realtime, tracking, forms, web, admin. +# +# One region on a container host, with the database, the root key and the +# object store in the cloud region next to it. Every value here is also what a +# joining node inherits, which is why the addresses have to be ones another +# machine can reach: `warmbly join` renders a node's configuration from this +# environment, and 127.0.0.1 does not resolve to your control plane from a VPS. + +APP_ENV=prod +DEPLOYMENT_MODE=self_hosted + +# --- data ------------------------------------------------------------------- +# Reached across the internet, so TLS is verified rather than merely offered. +PRIMARY_DB=postgres://warmbly:@db.eu-central-1.rds.amazonaws.com:5432/warmbly?sslmode=verify-full +REDIS=rediss://:@bus.example.com:6380 + +# --- event bus -------------------------------------------------------------- +EVENTBUS_PROVIDER=nats +NATS_URL=tls://@bus.example.com:4222 +# json is required wherever workers are exercised: the worker command and +# result envelopes carry bodies Avro cannot serialize. +CODEC_PROVIDER=json + +# --- encryption ------------------------------------------------------------- +# Nodes are handed KMS_PROVIDER=brokered instead of this, so no machine in the +# fleet needs an AWS credential of its own. +KMS_PROVIDER=aws +KMS_AWS_KEY_ID=alias/warmbly +AWS_REGION=eu-central-1 +AWS_ACCESS_KEY_ID= +AWS_SECRET_ACCESS_KEY= + +# Back these two up before storing a single mailbox. Losing either is +# unrecoverable, and a database backup without them cannot be decrypted. +AUTH_SECRET= +CREDENTIALS_ENCRYPTION_KEY= +INTERNAL_API_TOKEN= +SECRET_KEY_BASE= + +# --- storage ---------------------------------------------------------------- +# s3, not filesystem: a worker on another machine reads the message body the +# backend wrote, and it does not have your disk. +BLOB_PROVIDER=s3 +BLOB_BUCKET=warmbly-blobs- +BLOB_PUBLIC_BASE_URL=https://api.example.com/public + +# --- addresses -------------------------------------------------------------- +API_PUBLIC_URL=https://api.example.com +# What a node is told to call back on. It must be reachable from every machine +# in the fleet, not just from inside the container network. +ENCRYPTED_KEYS_BACKEND_URL=https://api.example.com +APP_URL=https://app.example.com +CORS_ALLOW_ORIGINS=https://app.example.com,https://admin.example.com +WEBSOCKET_URL=wss://rt.example.com/socket/websocket +PHX_HOST=rt.example.com +TRACKING_DOMAIN=track.example.com +FORMS_DOMAIN=forms.example.com +# The CIDRs your container host's proxy sends from. Without it every request +# looks like it came from the proxy and per-IP rate limiting protects nothing. +TRUSTED_PROXIES= + +# --- platform mail ---------------------------------------------------------- +# Login codes, invitations, resets and digests. NOT campaign mail, which goes +# out through the mailboxes your customers connect. +# +# SES needs a verified domain identity with published DKIM records, and +# production access: a sandboxed account only delivers to verified addresses. +# The consumer sends notification digests through the same transport, so it +# needs ses:SendEmail as well. +MAIL_TRANSPORT=ses +EMAIL_ADDRESS=noreply@example.com +EMAIL_NAME=Warmbly + +# --- mailbox OAuth clients -------------------------------------------------- +BOX_GOOGLE_CLIENT_ID= +BOX_GOOGLE_CLIENT_SECRET= +BOX_OUTLOOK_CLIENT_ID= +BOX_OUTLOOK_CLIENT_SECRET= diff --git a/deploy/split-cloud/node/docker-compose.yml b/deploy/split-cloud/node/docker-compose.yml new file mode 100644 index 00000000..88ce93e1 --- /dev/null +++ b/deploy/split-cloud/node/docker-compose.yml @@ -0,0 +1,26 @@ +# A worker box, run by hand instead of by `warmbly join`. +# +# The join script is the shorter path and gives you the auto-update timer: +# +# curl -fsSL https://api.example.com/join.sh | sh -s -- \ +# --url https://api.example.com --token \ +# --role worker --region eu-central +# +# Use this file when you want the node's configuration in version control, or +# when you run several workers per host. You keep the version current yourself +# with `docker compose pull && docker compose up -d`; `warmblyctl fleet version` +# has no effect on a node that was not joined. + +services: + worker: + image: ghcr.io/warmbly/warmbly/worker:${WARMBLY_TAG:-prod} + restart: unless-stopped + env_file: ./worker.env + volumes: + # The worker claims a stable id from this volume, so a recreated + # container keeps its mailboxes instead of registering as a new machine + # and leaving the old row holding them. + - worker_state:/data/state + +volumes: + worker_state: diff --git a/deploy/split-cloud/node/worker.env.example b/deploy/split-cloud/node/worker.env.example new file mode 100644 index 00000000..004f5e16 --- /dev/null +++ b/deploy/split-cloud/node/worker.env.example @@ -0,0 +1,56 @@ +# A worker on a machine you own, against a control plane elsewhere. +# +# Copy to worker.env and fill in. `warmbly join` writes the equivalent of this +# file for you; it is here so the hand-run path is not guesswork. +# +# What is NOT here is the point: no AWS access key, no bucket credential, no +# KMS key id, and no database DSN. A worker reaches relational data through the +# internal API, and the brokered providers below ask the control plane to +# perform the two operations that would otherwise need a cloud credential. + +APP_ENV=prod + +# One per machine, generated once: uuidgen. Keep it stable, because moving a +# mailbox to a different worker changes the client address its provider sees +# and buys a sign-in challenge for nothing. +WORKER_ID= +# Optional. Placement prefers a worker near where a mailbox's provider expects +# sign-ins. Blank is fine. +WARMBLY_NODE_REGION= + +# --- the control plane ------------------------------------------------------ +# One credential, for one instance, revocable from it. +ENCRYPTED_KEYS_PROVIDER=http +ENCRYPTED_KEYS_BACKEND_URL=https://api.example.com +ENCRYPTED_KEYS_WORKER_TOKEN= +INTERNAL_API_TOKEN= + +# --- the bus and cache ------------------------------------------------------ +# TLS on both: the bus carries recipient addresses and the cache carries each +# organization's decrypted data key. +EVENTBUS_PROVIDER=nats +NATS_URL=tls://@bus.example.com:4222 +CODEC_PROVIDER=json +REDIS=rediss://:@bus.example.com:6380 + +# --- crypto and storage ----------------------------------------------------- +# brokered: hold no key material and no bucket credential, and ask the control +# plane for the one privileged operation each. Blob bytes still travel directly +# between this machine and the object store. +KMS_PROVIDER=brokered +BLOB_PROVIDER=brokered + +# Seals mailbox SMTP and IMAP passwords and OAuth tokens at rest. The same +# value as the control plane, or nothing this machine reads will decrypt. +CREDENTIALS_ENCRYPTION_KEY= + +# --- mailbox OAuth clients -------------------------------------------------- +# Needed to refresh Gmail and Microsoft 365 tokens. Plain SMTP and IMAP +# mailboxes need none of it. +BOX_GOOGLE_CLIENT_ID= +BOX_GOOGLE_CLIENT_SECRET= +BOX_OUTLOOK_CLIENT_ID= +BOX_OUTLOOK_CLIENT_SECRET= + +# Real mailboxes present real certificates. +MAIL_TLS_INSECURE=false diff --git a/docs/content/docs/development/bare-metal.mdx b/docs/content/docs/development/bare-metal.mdx index d7b5dfa4..bc5fbf46 100644 --- a/docs/content/docs/development/bare-metal.mdx +++ b/docs/content/docs/development/bare-metal.mdx @@ -546,20 +546,21 @@ The point of workers is to spread sending across machine identities, so most ins **The join script**, which is what the admin panel's Add a machine flow produces, runs the node as a container on the remote host. It needs Docker and systemd only there. The backend serves it at `GET /join.sh`, and it is embedded in the binary, so there is no path to configure. -**A native worker** is the same binary and env file as above, with the addresses changed. The backend renders a complete env file for an enrollment token, so nothing has to be copied by hand: +**A native worker** is the same binary and env file as above, with the addresses changed. The backend renders a complete env file for a join token, so nothing has to be copied by hand: ```bash # On the remote host, with /opt/warmbly/bin/worker built for it: -curl -fsS https://api.yourdomain.com/api/v1/workers/enroll \ +curl -fsS https://api.yourdomain.com/api/v1/fleet/join \ -H 'Content-Type: application/json' \ - -d '{"token":"wmenroll_..."}' | sudo tee /etc/warmbly/worker.env >/dev/null -echo "WORKER_ID=$(uuidgen)" | sudo tee -a /etc/warmbly/worker.env >/dev/null + -d '{"token":"","role":"worker","region":"eu-central"}' \ + | sed -n 's/.*"env_b64"[[:space:]]*:[[:space:]]*"\([^"]*\)".*/\1/p' \ + | base64 -d | sudo tee /etc/warmbly/worker.env >/dev/null sudo chmod 0600 /etc/warmbly/worker.env sudo cp deploy/systemd/warmbly-worker.service /etc/systemd/system/ sudo systemctl daemon-reload && sudo systemctl enable --now warmbly-worker ``` -The token is consumed on first use and the response carries the decryption material, so do this over HTTPS only. Before enrolling anything, set `API_PUBLIC_URL`, `NATS_URL` and `REDIS` on the backend to addresses the remote host can reach, and open those ports to it: the rendered file inherits the backend's own values, and `127.0.0.1` does not resolve to your control plane from another machine. +The rendered file already carries this node's `WORKER_ID`, so its placement row and its node row are the same machine. Issue the token with `warmblyctl fleet join-token`; it is consumed on first use and the response carries the decryption material, so do this over HTTPS only. Before enrolling anything, set `API_PUBLIC_URL`, `NATS_URL` and `REDIS` on the backend to addresses the remote host can reach, and open those ports to it: the rendered file inherits the backend's own values, and `127.0.0.1` does not resolve to your control plane from another machine. Opening NATS and Redis to another machine means they stop being protected by the loopback bind, so give both a credential and TLS first. For NATS, extend `/etc/nats.conf` and restart it: diff --git a/docs/content/docs/development/configuration.mdx b/docs/content/docs/development/configuration.mdx index 53d9e61a..99032f60 100644 --- a/docs/content/docs/development/configuration.mdx +++ b/docs/content/docs/development/configuration.mdx @@ -240,12 +240,12 @@ Verdicts are re-checked after 90 days (30 for an inconclusive one), in passes of | Variable | What it does | Default | Restart needed | |---|---|---|---| -| `KMS_PROVIDER` | `local` (AES master key below) or `aws` (AWS KMS) | `local` under compose, `aws` for a bare binary | yes | +| `KMS_PROVIDER` | `local` (AES master key below), `aws` (AWS KMS), or `brokered` (hold no key material and ask the control plane to unwrap). A joining node is given `brokered` automatically wherever the instance runs `aws`, so no machine in the fleet needs a cloud credential | `local` under compose, `aws` for a bare binary | yes | | `KMS_LOCAL_MASTER_KEY` | base64, exactly 32 bytes. The root of trust for every per-organization data key | published default under compose | yes | | `KMS_LOCAL_MASTER_KEY_FILE` | Path to a file holding that key instead. Mutually exclusive with the inline value | unset | yes | | `KMS_AWS_KEY_ID` | Key id or alias when `KMS_PROVIDER=aws` | unset | yes | | `CREDENTIALS_ENCRYPTION_KEY` | exactly 64 hex characters. Seals mailbox SMTP and IMAP passwords at rest | published default under compose | yes | -| `ENCRYPTED_KEYS_PROVIDER` | `postgres` for backend and consumer, `http` for workers | the caller's fallback, so set it explicitly | yes | +| `ENCRYPTED_KEYS_PROVIDER` | `postgres` for the backend and a consumer that has a DSN, `http` for workers | the caller's fallback, so set it explicitly | yes | | `ENCRYPTED_KEYS_BACKEND_URL` | Where a worker reaches the backend's key endpoint | unset | yes | | `ENCRYPTED_KEYS_WORKER_TOKEN` | The worker's copy of `INTERNAL_API_TOKEN` | unset | yes | @@ -255,7 +255,7 @@ An empty `CREDENTIALS_ENCRYPTION_KEY` does not fail at boot. It disables sealing | Variable | What it does | Default | Restart needed | |---|---|---|---| -| `BLOB_PROVIDER` | `filesystem` or `s3` | `filesystem` under compose, `s3` for a bare binary | yes | +| `BLOB_PROVIDER` | `filesystem`, `s3`, or `brokered` (hold no bucket credential and ask the control plane to sign each operation; the bytes still travel directly). A joining node is given `brokered` automatically wherever the instance runs `s3` | `filesystem` under compose, `s3` for a bare binary | yes | | `BLOB_FS_ROOT` | Directory for stored bodies, attachments, avatars and email body images. The backend, the consumer and every worker on the host must share it | `/data/blobs` | yes | | `BLOB_BUCKET` | Bucket name when `BLOB_PROVIDER=s3` | unset | yes | | `BLOB_PUBLIC_BASE_URL` | Public base for the backend's `/public` route. Images placed in an email body are served from here, so it has to be an address a recipient's mail client can reach | derived | yes | @@ -333,6 +333,25 @@ An empty `ENCRYPTED_KEYS_BACKEND_URL` or `ENCRYPTED_KEYS_WORKER_TOKEN` lets the Workers hold no database connection by design. Everything relational they need arrives over the backend's internal HTTP API. +### What a joining node is given + +`warmbly join` renders `/etc/warmbly/node.env` from the backend's own environment, so most of this page reaches a node without being set twice. Three things are decided rather than copied: + +- **The crypto and blob providers are translated.** An instance on `KMS_PROVIDER=aws` hands its nodes `brokered`, and one on `BLOB_PROVIDER=s3` hands them `brokered` too, so no machine in the fleet carries a cloud credential. Providers that need no credential pass through unchanged. +- **A worker never receives `PRIMARY_DB`.** A consumer does, because it opens Postgres itself. An instance that keeps its DSN in a secret manager rather than the environment has none to send, and the join script says so. +- **Addresses are inherited literally.** `NATS_URL`, `REDIS` and `ENCRYPTED_KEYS_BACKEND_URL` are used exactly as the backend has them, and a value that only resolves inside your container network produces a node that enrols cleanly and reaches nothing. + +### Settings the control plane cannot know + +`node.env` is rewritten on every join. Next to it, `/etc/warmbly/node.local.env` is created once and never written again, and the container reads it second, so a name repeated there wins. + +That is where a per-machine value belongs: a DSN this instance does not hold, credentials for infrastructure of your own, or a deliberate override back to a direct provider. + +```bash +printf 'PRIMARY_DB=%s\n' "postgres://..." >> /etc/warmbly/node.local.env +systemctl restart warmbly-consumer +``` + ## Mailbox connections Needed on the backend **and** every worker: the backend starts the OAuth flow, and each worker refreshes the token when it expires. diff --git a/docs/content/docs/development/deployment-guide.mdx b/docs/content/docs/development/deployment-guide.mdx index 2ba4cffd..05bcdf31 100644 --- a/docs/content/docs/development/deployment-guide.mdx +++ b/docs/content/docs/development/deployment-guide.mdx @@ -727,7 +727,13 @@ The script enrols the node, writes the config the control plane hands back to `/ The config handed to a node is generated from the backend's own environment. On a stock local install that means `ENCRYPTED_KEYS_BACKEND_URL=http://localhost:8080`, `NATS_URL=nats://nats:4222`, and `REDIS=redis://redis:6379`, none of which resolve on another machine. Before adding a remote node, set `API_PUBLIC_URL`, `NATS_URL`, and `REDIS` on the backend to addresses the machine can actually reach, and open those ports. -That config carries the decryption material the node needs: the internal API token, `KMS_LOCAL_MASTER_KEY`, and `CREDENTIALS_ENCRYPTION_KEY`. Serve the API over HTTPS before adding a node across a network you do not control. It deliberately does not include `PRIMARY_DB`: a worker reaches relational data through the internal API and nothing else. +That config carries the decryption material the node needs: the internal API token, `KMS_LOCAL_MASTER_KEY`, and `CREDENTIALS_ENCRYPTION_KEY`. Serve the API over HTTPS before adding a node across a network you do not control. + +`PRIMARY_DB` reaches a consumer, which opens Postgres itself, and never a worker, which reaches relational data through the internal API and nothing else. + +Nothing else in the config needs a credential either. An instance running `KMS_PROVIDER=aws` or `BLOB_PROVIDER=s3` hands its nodes the brokered form of each, so a machine in the fleet opens sealed keys and signs blob operations through the internal API rather than carrying a cloud credential of its own. Blob bytes still go directly between the node and the object store. [Split deployment](/development/split-deployment/) covers the whole shape. + +`/etc/warmbly/node.local.env`, next to the generated file, is created once and never rewritten, and the container reads it second. Anything you add there survives a re-join and wins over the generated value. `BLOB_PROVIDER=filesystem` does not survive a fleet. A worker reads the message body the backend wrote, so the two need the same storage with permissions that let both reach it, and a node on another machine has neither. The join script creates and mounts `BLOB_FS_ROOT` so the node starts, and warns you, but sends will fail when the worker cannot read the body. Use `BLOB_PROVIDER=s3` with a bucket both sides can reach before running nodes off-host. diff --git a/docs/content/docs/development/instance-health.mdx b/docs/content/docs/development/instance-health.mdx index 57a27cbe..61ad2e31 100644 --- a/docs/content/docs/development/instance-health.mdx +++ b/docs/content/docs/development/instance-health.mdx @@ -217,6 +217,12 @@ No worker has checked in for more than five minutes while mailboxes are assigned Check the worker process is running, then check `ENCRYPTED_KEYS_BACKEND_URL` and `ENCRYPTED_KEYS_WORKER_TOKEN`: an empty value lets the worker start, subscribe and never register, with no log line to tell you. See [workers](/development/configuration/#workers). +### fleet_infra_unreachable + +Nodes are checking in from more than one machine, and `NATS_URL`, `REDIS` or `ENCRYPTED_KEYS_BACKEND_URL` still names a host that only resolves here, such as a compose service name or loopback. A joining node inherits those values verbatim, so it enrols, keeps heartbeating over HTTP, and reaches neither the bus nor the cache. The fleet looks healthy the whole time, which is why this is a check rather than a log line. See [split deployment](/development/split-deployment/). + +More than one machine is read as more than one distinct address among live nodes, so scaled replicas on a single host do not trigger it. + ### codec_not_json `CODEC_PROVIDER` is something other than `json` while workers are registered. Worker command and result envelopes carry untyped bodies that Avro cannot serialize, so every worker command fails to encode. Set `CODEC_PROVIDER=json`. See [event bus](/development/configuration/#event-bus). @@ -247,6 +253,10 @@ A reachable realtime service that still leaves the dashboard dead has a differen `BLOB_PROVIDER=filesystem` and `BLOB_FS_ROOT` is unset, missing or not writable, so email bodies, attachments and avatars cannot be stored. The check writes and deletes a probe file to prove the path is usable. The backend, the consumer and every worker on the host must share that path. See [storage](/development/configuration/#storage). +### fleet_blobs_not_shared + +`BLOB_PROVIDER=filesystem` while nodes check in from more than one machine. A worker reads the message body the backend wrote, so a node that does not share this filesystem cannot send, and nothing says so until the last step of a send. Move to `BLOB_PROVIDER=s3` with a bucket both sides reach. See [split deployment](/development/split-deployment/). + ## What each service needs to be healthy | Service | Needs | diff --git a/docs/content/docs/development/meta.json b/docs/content/docs/development/meta.json index 15ad7489..d774963f 100644 --- a/docs/content/docs/development/meta.json +++ b/docs/content/docs/development/meta.json @@ -7,6 +7,7 @@ "install", "deployment-guide", "bare-metal", + "split-deployment", "first-run", "accounts-and-access", "admin-panel", diff --git a/docs/content/docs/development/split-deployment.mdx b/docs/content/docs/development/split-deployment.mdx new file mode 100644 index 00000000..ece297cd --- /dev/null +++ b/docs/content/docs/development/split-deployment.mdx @@ -0,0 +1,200 @@ +--- +title: Split deployment +description: Running the control plane on a container host, the fleet on machines you own, and the database, root key and object store in a cloud region, with what each piece costs. +--- + +The [self-hosting guide](/development/deployment-guide/) puts everything on one machine, which is the right answer until it is not. This page is the other shape: the control plane on a container host, the worker fleet on machines you own, and the durable pieces in a cloud region next to the control plane. + +It is the deployment Warmbly's own architecture describes. The control plane owns relational state and runs where a container host is convenient. The execution plane is a fleet of machines you rent, because a worker's value is being a distinct machine somewhere. + +``` +container host machines you own cloud region +────────────── ──────────────── ──────────── +backend worker Postgres +consumer worker KMS key +realtime nats + redis S3 bucket +tracking SES +forms +web / admin +``` + +Everything here is env-var configuration and manifests in [`deploy/split-cloud/`](https://github.com/warmbly/warmbly/tree/main/deploy/split-cloud). No code changes, no fork. + +## What goes where, and why + +| Service | Plane | Why there | +|---------|-------|-----------| +| backend | control | Owns Postgres and applies migrations on boot | +| consumer | control | Opens Postgres directly, so it belongs next to it | +| realtime | control | Websocket fanout; needs Redis and a public hostname | +| tracking | control | Public pixel and click endpoints on their own domain | +| forms | control | Only if you use hosted forms | +| web, admin | control | Static builds, configured at container start | +| worker | execution | Sends and syncs from machines you control | +| NATS, Redis | execution side | Both planes reach them; keeping them near the fleet keeps the noisy hop local | +| Postgres, KMS, S3, SES | cloud | Durability, a root of trust, and shared blob storage | + +The consumer is the one people put in the wrong place. It is not a worker: it updates relational state, and it needs the database DSN a worker is deliberately never given. Running it beside the backend saves a database connection crossing the internet for every event. + +## Keep the three regions close + +The backend reaches Postgres and Redis on every request, so the round trips between the three providers are the latency floor of the whole product. Pick a container-host region, a cloud region, and a datacentre for the machines that are near each other, and confirm it before you build anything on top. + +## Two things that are not optional + + +The cache holds each organization's decrypted data key for the life of its entry. A plaintext connection across the internet publishes key material, and a password does not change that. On one machine the loopback bind was the protection; once the control plane is somewhere else, TLS is. + + +**Blobs have to be object storage.** A worker reads the message body the backend wrote. With `BLOB_PROVIDER=filesystem` it has neither the disk nor the permissions, and the failure arrives at the last step of a send with everything else looking healthy. `warmbly join` warns about this and starts the node anyway; the warning is the whole warning you get. + +## Nodes carry no cloud credentials + +A machine in the fleet needs two privileged things: opening the sealed data key for an organization, and reading and writing message bodies. Both would normally mean an AWS credential on every box, in a file `warmbly join` rewrites on each run. + +It does not work that way. When the control plane runs `KMS_PROVIDER=aws` or `BLOB_PROVIDER=s3`, a joining node is handed the brokered form instead: + +| Control plane | What a node is given | +|---|---| +| `KMS_PROVIDER=aws` | `KMS_PROVIDER=brokered` | +| `KMS_PROVIDER=local` | `KMS_PROVIDER=local`, with the master key | +| `BLOB_PROVIDER=s3` | `BLOB_PROVIDER=brokered` | +| `BLOB_PROVIDER=filesystem` | `BLOB_PROVIDER=filesystem`, and a warning | + +A brokered provider authenticates with the internal API token the node already holds and asks the instance to perform the one privileged operation: unwrap this key, sign this blob operation. The instance token is scoped to one deployment and revocable from it, which an IAM access key on a rented machine is not. + +Blob bytes are not proxied. The control plane signs a URL and the node transfers directly against the object store, so a mailbox sync costs the backend one small request per object rather than the bytes. + +The cost is one HTTPS call per DEK open, which Redis caches, and one per blob operation. If you would rather a node talk to AWS directly, put `KMS_PROVIDER`, `BLOB_PROVIDER` and the credentials in `/etc/warmbly/node.local.env`, which a re-join does not overwrite. + +## Building it + + + + + +### The cloud resources + +```bash +scripts/aws-bootstrap.sh --domain example.com --region eu-central-1 +``` + +That creates the KMS key and alias, a private bucket with encryption on, a Postgres instance, an SES domain identity, and an IAM user for the control plane with a least-privilege policy. It is idempotent, and `--dry-run` prints what it would do. + +Five things it deliberately leaves to you, because each is a decision rather than a default: an access key for the control-plane user, the database security group, `rds.force_ssl=1` in the parameter group, the DKIM records in DNS, and SES production access. + + +Until you request production access, SES only delivers to addresses you have verified, which means invitations and password resets silently reach nobody else. This is platform mail only: campaign and warmup mail goes out through the mailboxes your customers connect, and never touches SES. + + + + + + +### The bus and cache + +On the machine that will run them, with a certificate for its hostname: + +```bash +cp -r deploy/split-cloud/bus /opt/warmbly/bus +cd /opt/warmbly/bus +printf 'NATS_TOKEN=%s\nREDIS_PASSWORD=%s\n' \ + "$(openssl rand -hex 32)" "$(openssl rand -hex 32)" > .env +docker compose up -d +``` + +`certbot-deploy-hook.sh` belongs in `/etc/letsencrypt/renewal-hooks/deploy/`. Neither service re-reads its certificate, so a renewal without it leaves both serving an expired one. + +Redis publishes only its TLS port. The plaintext port stays on the container network for anything co-located. + + + + + +### The control plane + +`deploy/split-cloud/control-plane.env.example` is every setting, annotated. The ones that decide whether a fleet can exist at all: + +```bash +PRIMARY_DB=postgres://warmbly:...@db.eu-central-1.rds.amazonaws.com:5432/warmbly?sslmode=verify-full +REDIS=rediss://:@bus.example.com:6380 +NATS_URL=tls://@bus.example.com:4222 +ENCRYPTED_KEYS_BACKEND_URL=https://api.example.com +BLOB_PROVIDER=s3 +``` + + +`warmbly join` renders a node's configuration from the backend's own environment. A value that only resolves inside your container network produces a node that enrols cleanly and then cannot reach anything. Set these to addresses another machine can use before adding one. + + +Deploy the backend first: it applies the migrations. Then the consumer, realtime, tracking, and the two frontends. + + + + + +### The fleet + +```bash +warmblyctl fleet join-token + +# on each machine, as root +curl -fsSL https://api.example.com/join.sh | sh -s -- \ + --url https://api.example.com \ + --token \ + --role worker \ + --region eu-central +``` + +Nothing connects back to the machine, then or later. It needs no inbound port and no SSH key, and re-running the same command re-joins it under the same identity, keeping its mailboxes. + +`deploy/split-cloud/node/` has the same thing as a compose file, for when you want a node's configuration in version control. A node run that way keeps itself current with `docker compose pull`; `warmblyctl fleet version` only moves nodes that were joined. + + + + + +## Anything the control plane cannot know + +`warmbly join` writes `/etc/warmbly/node.env` from the control plane's answer and rewrites it on every join. Next to it, `/etc/warmbly/node.local.env` is created once and never written again, and the container reads it second, so a name repeated there wins. + +That is the place for a value this instance does not hold: a DSN kept in a secret manager rather than the environment, credentials for infrastructure of your own, a per-machine tuning knob. + +```bash +printf 'PRIMARY_DB=%s\n' "postgres://..." >> /etc/warmbly/node.local.env +systemctl restart warmbly-consumer +``` + +## Sizing the fleet + +A worker's capacity is 16 cold-mailbox equivalents, and each mailbox declares its own weight: an SMTP or IMAP mailbox costs `1.0`, a Gmail or Outlook mailbox `0.05`, a warmup-only mailbox `0.4`. + +So one worker holds 16 SMTP mailboxes or around 300 OAuth ones. OAuth-heavy customers cost almost nothing in machines; SMTP and IMAP customers are what drive the count. + +Give NATS and Redis their own machine once you have more than one worker. Until then the control plane depends on a box that also runs a worker, and restarting that worker takes the bus with it. + +## What it costs + +A starting instance, with one machine running the bus and the first worker: + +| | Item | Monthly | +|---|---|---| +| Machines | One small VPS: bus, cache, first worker | ~$5 | +| Cloud | Postgres, smallest burstable instance with 20 GB | $13-15 | +| | KMS: one key plus requests, which Redis caches away | ~$1 | +| | Object storage: a few GB | ~$0.20 | +| | Platform mail: hundreds of messages | ~$0.05 | +| Container host | The control plane's actual usage | $5-10 | +| | **Total** | **~$25-32** | + +Two things move that number. A cloud account under twelve months old gets the smallest database instance free, which takes roughly $14 off. Turning on a standby doubles the database line, so leave it off at the start and take snapshots instead. + +The line that grows quietly is egress. Every query result and every message body a node reads crosses the internet, billed past the first 100 GB a month. It is invisible at launch and becomes the second-largest cloud line once mailbox sync is moving real volume. + +## Where things run out + +**One bus machine is one failure domain.** Nothing sends while it is down. Snapshots and a documented rebuild are the honest answer at this size; NATS clustering is the answer when the instance is worth more than the afternoon it costs. + +**The database is reachable from the internet.** That is what makes a container host and a fleet able to share it. Restrict the security group to the addresses that need it, force TLS, and treat the master password as the credential it is. + +**Auto-update covers nodes, not the control plane.** The backend is what tells every node which version to be, so it upgrades the way the rest of your container host does. Upgrade it, and the fleet follows. diff --git a/internal/api/handler/fleet_nodes.go b/internal/api/handler/fleet_nodes.go index c78293b7..f9136033 100644 --- a/internal/api/handler/fleet_nodes.go +++ b/internal/api/handler/fleet_nodes.go @@ -14,6 +14,7 @@ import ( "github.com/gin-gonic/gin" "github.com/google/uuid" "github.com/warmbly/warmbly/internal/app/fleetnode" + "github.com/warmbly/warmbly/internal/config" "github.com/warmbly/warmbly/internal/errx" "github.com/warmbly/warmbly/internal/models" ) @@ -198,9 +199,20 @@ func nodeHeartbeatSeconds(livenessSeconds int) int { // is configured with exactly the infrastructure the control plane uses and // there is nothing to keep in sync by hand. // -// Deliberately excluded: PRIMARY_DB and anything else that would give a node -// direct database access. Workers reach relational data through the internal -// API and nothing else, and handing them a DSN here would quietly undo that. +// Every name here must be one the node's own code reads. Two of them were not +// (S3_BUCKET and KMS_KEY_ID, against a storage layer reading BLOB_BUCKET and a +// KMS factory reading KMS_AWS_KEY_ID), which sent an AWS-backed node to the +// default bucket and the default key alias with nothing logged. +// +// KMS_PROVIDER and BLOB_PROVIDER are absent because they are not copied but +// translated; see nodeProviders. The AWS-shaped settings above stay, so an +// operator who deliberately overrides a node back to a direct provider in +// node.local.env only has to add the credential. +// +// Deliberately excluded: PRIMARY_DB for a worker, and anything else that would +// give one direct database access. A worker reaches relational data through +// the internal API and nothing else. renderNodeEnv sends the DSN to a consumer, +// which is control plane and updates relational state itself. var nodeEnvKeys = []string{ "APP_ENV", "EVENTBUS_PROVIDER", @@ -213,14 +225,13 @@ var nodeEnvKeys = []string{ "SCHEMA_REGISTRY_SECRET", "CODEC_PROVIDER", "REDIS", - "KMS_PROVIDER", "KMS_LOCAL_MASTER_KEY", - "KMS_KEY_ID", + "KMS_AWS_KEY_ID", "CREDENTIALS_ENCRYPTION_KEY", - "BLOB_PROVIDER", "BLOB_FS_ROOT", + "BLOB_BUCKET", "AWS_REGION", - "S3_BUCKET", + "AWS_ENDPOINT_URL_S3", "BOX_GOOGLE_CLIENT_ID", "BOX_GOOGLE_CLIENT_SECRET", "BOX_OUTLOOK_CLIENT_ID", @@ -229,6 +240,32 @@ var nodeEnvKeys = []string{ "SENTRY_DSN", } +// nodeProviders translates the control plane's own crypto and blob providers +// into the ones a node should run. +// +// A node has no cloud credentials and no way to be handed any: node.env is +// regenerated from this instance's environment on every join, and shipping an +// IAM key to every machine in the fleet is exactly the thing worth avoiding. So +// a provider that needs a credential becomes its brokered form, which +// authenticates with the internal API token the node already holds and asks +// this instance to do the one privileged operation. A provider that needs no +// credential (local KMS, filesystem blobs) passes through unchanged. +// +// Brokering costs one HTTPS call to the control plane per DEK open (Redis +// caches the result) and per blob operation. The bytes still go straight +// between the node and the object store. +func nodeProviders() (kmsProvider, blobProvider string) { + kmsProvider = config.KMSProvider() + if kmsProvider == "aws" || kmsProvider == "aws-kms" { + kmsProvider = "brokered" + } + blobProvider = config.BlobProvider() + if blobProvider == "s3" { + blobProvider = "brokered" + } + return kmsProvider, blobProvider +} + // renderNodeEnv builds the env file a node writes to disk on join. func renderNodeEnv(nodeID uuid.UUID, role models.NodeRole, region string) string { var b strings.Builder @@ -248,12 +285,29 @@ func renderNodeEnv(nodeID uuid.UUID, role models.NodeRole, region string) string if backend == "" { backend = strings.TrimRight(os.Getenv("APP_INTERNAL_URL"), "/") } + // A consumer is control plane: it opens Postgres itself, so it gets the DSN + // a worker is deliberately never given, and reads keys straight from the + // table rather than back through the API it sits behind. An instance whose + // DSN lives in SSM rather than the environment has nothing to send, so the + // node falls back to the worker's HTTP key path and join.sh tells the + // operator to supply PRIMARY_DB in node.local.env. + dsn := os.Getenv("PRIMARY_DB") + keysProvider := "http" + if role == models.NodeRoleConsumer && dsn != "" { + keysProvider = "postgres" + fmt.Fprintf(&b, "PRIMARY_DB=%s\n", dsn) + } + fmt.Fprintf(&b, "WARMBLY_BACKEND_URL=%s\n", backend) - fmt.Fprintf(&b, "ENCRYPTED_KEYS_PROVIDER=%s\n", "http") + fmt.Fprintf(&b, "ENCRYPTED_KEYS_PROVIDER=%s\n", keysProvider) fmt.Fprintf(&b, "ENCRYPTED_KEYS_BACKEND_URL=%s\n", backend) fmt.Fprintf(&b, "ENCRYPTED_KEYS_WORKER_TOKEN=%s\n", os.Getenv("INTERNAL_API_TOKEN")) fmt.Fprintf(&b, "INTERNAL_API_TOKEN=%s\n", os.Getenv("INTERNAL_API_TOKEN")) + kmsProvider, blobProvider := nodeProviders() + fmt.Fprintf(&b, "KMS_PROVIDER=%s\n", kmsProvider) + fmt.Fprintf(&b, "BLOB_PROVIDER=%s\n", blobProvider) + for _, k := range nodeEnvKeys { if v := os.Getenv(k); v != "" { fmt.Fprintf(&b, "%s=%s\n", k, v) diff --git a/internal/api/handler/fleet_nodes_test.go b/internal/api/handler/fleet_nodes_test.go new file mode 100644 index 00000000..fc4e9b49 --- /dev/null +++ b/internal/api/handler/fleet_nodes_test.go @@ -0,0 +1,176 @@ +package handler + +import ( + "strings" + "testing" + + "github.com/google/uuid" + "github.com/warmbly/warmbly/internal/models" +) + +// envLines turns a rendered node env into a lookup, so assertions name a +// setting rather than a line number. +func envLines(t *testing.T, rendered string) map[string]string { + t.Helper() + out := map[string]string{} + for _, line := range strings.Split(rendered, "\n") { + if line == "" || strings.HasPrefix(line, "#") { + continue + } + k, v, ok := strings.Cut(line, "=") + if !ok { + t.Fatalf("line is not KEY=value: %q", line) + } + out[k] = v + } + return out +} + +// setInstanceEnv puts the process in the shape of an AWS-backed control plane. +func setInstanceEnv(t *testing.T) { + t.Helper() + t.Setenv("ENCRYPTED_KEYS_BACKEND_URL", "https://api.example.com/") + t.Setenv("INTERNAL_API_TOKEN", "tok") + t.Setenv("KMS_PROVIDER", "aws") + t.Setenv("KMS_AWS_KEY_ID", "alias/warmbly") + t.Setenv("BLOB_PROVIDER", "s3") + t.Setenv("BLOB_BUCKET", "warmbly-blobs") + t.Setenv("AWS_REGION", "eu-central-1") + t.Setenv("CREDENTIALS_ENCRYPTION_KEY", "deadbeef") + t.Setenv("NATS_URL", "tls://bus.example.com:4222") + t.Setenv("REDIS", "rediss://bus.example.com:6380") +} + +// A node has no cloud credential and no way to be given one, so a provider +// that needs a credential has to arrive translated. Shipping "aws" here is how +// a joined node ends up unable to open a single mailbox. +func TestRenderNodeEnvBrokersCredentialProviders(t *testing.T) { + setInstanceEnv(t) + env := envLines(t, renderNodeEnv(uuid.New(), models.NodeRoleWorker, "eu-central")) + + if env["KMS_PROVIDER"] != "brokered" { + t.Errorf("KMS_PROVIDER = %q, want brokered", env["KMS_PROVIDER"]) + } + if env["BLOB_PROVIDER"] != "brokered" { + t.Errorf("BLOB_PROVIDER = %q, want brokered", env["BLOB_PROVIDER"]) + } + if _, ok := env["AWS_ACCESS_KEY_ID"]; ok { + t.Error("a node was handed an AWS credential") + } +} + +// A provider that needs no credential works on a node as it stands, and +// translating it would break a local install for nothing. +func TestRenderNodeEnvPassesThroughLocalProviders(t *testing.T) { + setInstanceEnv(t) + t.Setenv("KMS_PROVIDER", "local") + t.Setenv("BLOB_PROVIDER", "filesystem") + t.Setenv("BLOB_FS_ROOT", "/data/blobs") + + env := envLines(t, renderNodeEnv(uuid.New(), models.NodeRoleWorker, "")) + if env["KMS_PROVIDER"] != "local" { + t.Errorf("KMS_PROVIDER = %q, want local", env["KMS_PROVIDER"]) + } + if env["BLOB_PROVIDER"] != "filesystem" { + t.Errorf("BLOB_PROVIDER = %q, want filesystem", env["BLOB_PROVIDER"]) + } + if env["BLOB_FS_ROOT"] != "/data/blobs" { + t.Errorf("BLOB_FS_ROOT = %q", env["BLOB_FS_ROOT"]) + } +} + +// The regression this file exists for: every name sent has to be one the +// node's own code reads. S3_BUCKET and KMS_KEY_ID were read by nothing, so a +// node fell back to the default bucket and the default key alias in silence. +func TestRenderNodeEnvSendsNamesTheNodeReads(t *testing.T) { + setInstanceEnv(t) + t.Setenv("S3_BUCKET", "should-not-travel") + t.Setenv("KMS_KEY_ID", "should-not-travel") + + env := envLines(t, renderNodeEnv(uuid.New(), models.NodeRoleWorker, "")) + for _, dead := range []string{"S3_BUCKET", "KMS_KEY_ID"} { + if _, ok := env[dead]; ok { + t.Errorf("%s is still sent; nothing reads it", dead) + } + } + if env["BLOB_BUCKET"] != "warmbly-blobs" { + t.Errorf("BLOB_BUCKET = %q, want warmbly-blobs", env["BLOB_BUCKET"]) + } + if env["KMS_AWS_KEY_ID"] != "alias/warmbly" { + t.Errorf("KMS_AWS_KEY_ID = %q, want alias/warmbly", env["KMS_AWS_KEY_ID"]) + } +} + +// A worker reaches relational data through the internal API and nothing else. +// The DSN must not travel even when the control plane has one to send. +func TestRenderNodeEnvWithholdsDSNFromWorker(t *testing.T) { + setInstanceEnv(t) + t.Setenv("PRIMARY_DB", "postgres://u:p@db/warmbly") + + env := envLines(t, renderNodeEnv(uuid.New(), models.NodeRoleWorker, "")) + if _, ok := env["PRIMARY_DB"]; ok { + t.Fatal("a worker was handed a database DSN") + } + if env["ENCRYPTED_KEYS_PROVIDER"] != "http" { + t.Errorf("ENCRYPTED_KEYS_PROVIDER = %q, want http", env["ENCRYPTED_KEYS_PROVIDER"]) + } +} + +// A consumer is control plane: it opens Postgres itself and cannot boot +// without the DSN, which is why joining one used to produce a node that died +// on its first start. +func TestRenderNodeEnvGivesConsumerTheDSN(t *testing.T) { + setInstanceEnv(t) + t.Setenv("PRIMARY_DB", "postgres://u:p@db/warmbly") + + id := uuid.New() + env := envLines(t, renderNodeEnv(id, models.NodeRoleConsumer, "")) + if env["PRIMARY_DB"] != "postgres://u:p@db/warmbly" { + t.Errorf("PRIMARY_DB = %q", env["PRIMARY_DB"]) + } + if env["ENCRYPTED_KEYS_PROVIDER"] != "postgres" { + t.Errorf("ENCRYPTED_KEYS_PROVIDER = %q, want postgres", env["ENCRYPTED_KEYS_PROVIDER"]) + } + // Only a worker claims a placement identity. + if _, ok := env["WORKER_ID"]; ok { + t.Error("a consumer was given a WORKER_ID") + } + if env["WARMBLY_NODE_ID"] != id.String() { + t.Errorf("WARMBLY_NODE_ID = %q, want %s", env["WARMBLY_NODE_ID"], id) + } +} + +// An instance holding its DSN in SSM has none to send. The node still has to +// be able to fetch keys, so it falls back to the HTTP path rather than being +// left with a provider it cannot satisfy. +func TestRenderNodeEnvConsumerWithoutDSNFallsBackToHTTP(t *testing.T) { + setInstanceEnv(t) + t.Setenv("PRIMARY_DB", "") + + env := envLines(t, renderNodeEnv(uuid.New(), models.NodeRoleConsumer, "")) + if _, ok := env["PRIMARY_DB"]; ok { + t.Error("PRIMARY_DB was sent as an empty value") + } + if env["ENCRYPTED_KEYS_PROVIDER"] != "http" { + t.Errorf("ENCRYPTED_KEYS_PROVIDER = %q, want http", env["ENCRYPTED_KEYS_PROVIDER"]) + } +} + +// The worker's identity and the node row have to be the same machine. +func TestRenderNodeEnvWorkerIdentity(t *testing.T) { + setInstanceEnv(t) + id := uuid.New() + env := envLines(t, renderNodeEnv(id, models.NodeRoleWorker, "eu-central")) + + if env["WORKER_ID"] != id.String() { + t.Errorf("WORKER_ID = %q, want %s", env["WORKER_ID"], id) + } + if env["WARMBLY_NODE_REGION"] != "eu-central" { + t.Errorf("WARMBLY_NODE_REGION = %q", env["WARMBLY_NODE_REGION"]) + } + // The trailing slash on the instance URL must not survive into a base URL + // the node concatenates paths onto. + if env["ENCRYPTED_KEYS_BACKEND_URL"] != "https://api.example.com" { + t.Errorf("ENCRYPTED_KEYS_BACKEND_URL = %q", env["ENCRYPTED_KEYS_BACKEND_URL"]) + } +} diff --git a/internal/api/handler/handler.go b/internal/api/handler/handler.go index 138b14c4..9ac32894 100644 --- a/internal/api/handler/handler.go +++ b/internal/api/handler/handler.go @@ -72,6 +72,7 @@ import ( "github.com/warmbly/warmbly/internal/pkg/generation" "github.com/warmbly/warmbly/internal/infrastructure/encryptedkeys" + "github.com/warmbly/warmbly/internal/infrastructure/kms" "github.com/warmbly/warmbly/internal/infrastructure/pubsub" "github.com/warmbly/warmbly/internal/infrastructure/storage" "github.com/warmbly/warmbly/internal/models" @@ -275,6 +276,12 @@ type Handler struct { // HTTP-proxy implementation. EncryptedKeys encryptedkeys.Store + // The instance's root of trust, used by /api/v1/internal/dek/decrypt to + // open a sealed key for a node that holds no KMS credential of its own. + // Nothing else in the handler layer touches it: application crypto goes + // through the cipher service. + KMS kms.Provider + // Worker messageId -> internal email map, served to workers over HTTPS at // /api/v1/internal/email-message-map for the same no-direct-Postgres reason // as EncryptedKeys. Backed by Postgres in the backend. diff --git a/internal/api/handler/internal_blobs.go b/internal/api/handler/internal_blobs.go new file mode 100644 index 00000000..5eaf53e8 --- /dev/null +++ b/internal/api/handler/internal_blobs.go @@ -0,0 +1,84 @@ +package handler + +import ( + "errors" + "net/http" + "time" + + "github.com/gin-gonic/gin" + "github.com/warmbly/warmbly/internal/infrastructure/storage" +) + +// Internal blob brokering, used by nodes that hold no credential for the +// object store. The control plane signs one operation on one key; the node +// then talks to the store directly, so message bodies and attachments never +// pass through the backend and it pays no bandwidth for a mailbox sync. +// +// Auth is middleware.InternalAuthMiddleware, the same INTERNAL_API_TOKEN every +// other internal route uses. +// +// POST /api/v1/internal/blobs/presign +// body {"op":"get|put|head|delete","key":"...","content_type":"..."} +// 200 {"url":"...","method":"GET","expires_in":300} +// 501 the instance's blob backend cannot sign (filesystem) + +// blobPresignTTL is how long a signed URL lives. Long enough for a large +// attachment over a slow link, short enough that one captured in a log is +// already dead. +const blobPresignTTL = 5 * time.Minute + +type blobPresignRequest struct { + Op string `json:"op"` + Key string `json:"key"` + ContentType string `json:"content_type"` +} + +type blobPresignResponse struct { + URL string `json:"url"` + Method string `json:"method"` + ExpiresIn int `json:"expires_in"` +} + +// InternalPresignBlob signs one blob operation for a node. +func (h *Handler) InternalPresignBlob(c *gin.Context) { + if h.Storage == nil { + c.JSON(http.StatusNotImplemented, gin.H{"error": "no blob store configured"}) + return + } + var req blobPresignRequest + if err := c.ShouldBindJSON(&req); err != nil { + c.JSON(http.StatusBadRequest, gin.H{"error": "decode body"}) + return + } + op := storage.PresignOp(req.Op) + if !op.Valid() { + c.JSON(http.StatusBadRequest, gin.H{"error": "op must be one of get, put, head, delete"}) + return + } + if req.Key == "" { + c.JSON(http.StatusBadRequest, gin.H{"error": "key required"}) + return + } + + url, err := h.Storage.PresignedURL(c.Request.Context(), op, req.Key, req.ContentType, blobPresignTTL) + if err != nil { + // The filesystem backend cannot sign anything, and a node on another + // machine could not reach those bytes even if it could: they are on a + // disk it does not have. Say which it is, because the fix is a + // different BLOB_PROVIDER rather than a retry. + if errors.Is(err, storage.ErrUnsupported) { + c.JSON(http.StatusNotImplemented, gin.H{ + "error": "this instance's blob backend cannot sign URLs; a fleet needs BLOB_PROVIDER=s3", + }) + return + } + c.JSON(http.StatusInternalServerError, gin.H{"error": "could not sign blob url"}) + return + } + + c.JSON(http.StatusOK, blobPresignResponse{ + URL: url, + Method: op.Method(), + ExpiresIn: int(blobPresignTTL.Seconds()), + }) +} diff --git a/internal/api/handler/internal_dek.go b/internal/api/handler/internal_dek.go index 7462e17d..574f9070 100644 --- a/internal/api/handler/internal_dek.go +++ b/internal/api/handler/internal_dek.go @@ -1,6 +1,7 @@ package handler import ( + "encoding/base64" "encoding/json" "errors" "io" @@ -21,11 +22,17 @@ import ( // PUT /api/v1/internal/dek/:orgID body: {"encrypted_data_key":"..."} // -> 201 | 409 ErrAlreadyExists // DELETE /api/v1/internal/dek/:orgID -> 204 +// POST /api/v1/internal/dek/decrypt body: {"encrypted_data_key":"..."} +// -> 200 {"data_key":""} type dekPayload struct { EncryptedDataKey string `json:"encrypted_data_key"` } +type dekDecryptResponse struct { + DataKey string `json:"data_key"` +} + func parseOrgID(c *gin.Context) (uuid.UUID, bool) { id, err := uuid.Parse(c.Param("orgID")) if err != nil { @@ -93,3 +100,39 @@ func (h *Handler) InternalDeleteDEK(c *gin.Context) { } c.Status(http.StatusNoContent) } + +// InternalDecryptDEK opens a sealed data key for a node, so the node needs no +// credential for the KMS behind it. +// +// Without this a worker running against AWS KMS needs its own IAM credential +// good for kms:Decrypt, which means a long-lived access key on every machine +// in the fleet. This endpoint replaces that with the internal API token the +// node already holds, which is scoped to this instance and revocable from it. +// +// It takes ciphertext and returns plaintext, with no organization id anywhere +// in the exchange: a caller can only open a key it was already given, so this +// grants nothing beyond what holding the sealed key and the token already did. +func (h *Handler) InternalDecryptDEK(c *gin.Context) { + if h.KMS == nil { + c.JSON(http.StatusNotImplemented, gin.H{"error": "no kms provider configured"}) + return + } + var p dekPayload + if err := c.ShouldBindJSON(&p); err != nil { + c.JSON(http.StatusBadRequest, gin.H{"error": "decode body"}) + return + } + if p.EncryptedDataKey == "" { + c.JSON(http.StatusBadRequest, gin.H{"error": "encrypted_data_key required"}) + return + } + key, err := h.KMS.GetDecryptedKey(c.Request.Context(), p.EncryptedDataKey) + if err != nil { + // The reason is not echoed: this answers an unauthenticated-by-org + // caller, and KMS errors distinguish "not a key of ours" from "malformed", + // which is exactly what a prober wants to learn. + c.JSON(http.StatusBadRequest, gin.H{"error": "could not decrypt data key"}) + return + } + c.JSON(http.StatusOK, dekDecryptResponse{DataKey: base64.StdEncoding.EncodeToString(key)}) +} diff --git a/internal/api/handler/nodescript/join.sh b/internal/api/handler/nodescript/join.sh index 10b0f827..91e8d03b 100755 --- a/internal/api/handler/nodescript/join.sh +++ b/internal/api/handler/nodescript/join.sh @@ -57,7 +57,9 @@ Join a machine to a Warmbly fleet. -h, --help This text. A second run re-enrols the same machine: it keeps the existing node id, so the -node keeps its identity, history and mailbox placements. +node keeps its identity, history and mailbox placements. It rewrites node.env +from the control plane's answer; put anything of your own in node.local.env +next to it, which is created once and never written again. USAGE } @@ -178,6 +180,9 @@ write_config() { log "--dry-run: would write $CONFIG_DIR/node.env with:" printf '%s\n' "$NODE_ENV" | sed 's/\(TOKEN=\|KEY=\|SECRET=\|PASSWORD=\).*/\1***/' | sed 's/^/ /' log "" + log "--dry-run: would create $CONFIG_DIR/node.local.env if absent, and leave it" + log " alone if present. Both files are passed to the container." + log "" log "--dry-run: would run image $WARMBLY_IMAGE_REPO/$WARMBLY_ROLE:$DESIRED_VERSION" return 0 fi @@ -216,6 +221,8 @@ write_config() { } > "$CONFIG_DIR/node.env" chmod 600 "$CONFIG_DIR/node.env" + ensure_local_env + printf '%s\n' "$DESIRED_VERSION" > "$AGENT_DIR/target-version" chown 1000:1000 "$AGENT_DIR/target-version" 2>/dev/null || true printf '%s\n' "$WARMBLY_IMAGE_REPO/$WARMBLY_ROLE" > "$STATE_DIR/image" @@ -226,6 +233,32 @@ write_config() { log "Wrote $CONFIG_DIR/node.env" } +# ensure_local_env creates the operator's own env file, once. node.env is +# rewritten wholesale on every join, so anything added there is lost the next +# time this runs; this file is the place that survives. The container reads +# both, this one second, so a value here wins. +# +# Never truncates an existing file: re-running a join must not discard the +# credential someone put here. +ensure_local_env() { + if [ -f "$CONFIG_DIR/node.local.env" ]; then + return 0 + fi + cat > "$CONFIG_DIR/node.local.env" <<'LOCALENV' +# Local overrides for this machine, read after node.env, so a name repeated +# here wins. +# +# `warmbly join` creates this file once and never writes it again, which makes +# it the place for anything the control plane cannot know: a DSN it does not +# hold, credentials for infrastructure of your own, a per-machine tuning knob. +# +# KEY=value, one per line, no export, no quotes needed. +LOCALENV + chmod 600 "$CONFIG_DIR/node.local.env" + log "Created $CONFIG_DIR/node.local.env (yours; re-joining leaves it alone)" + return 0 +} + # Blob handling. All of it reads the env in memory rather than the file on # disk, so --dry-run reports the same thing a real join would do instead of # reading a previous join's leftovers or failing on a file that is not there. @@ -340,6 +373,30 @@ warn_shared_blobs() { warn "" } +# warn_missing_db covers the one config a consumer cannot start without and the +# control plane cannot always supply: an instance holding its DSN in SSM rather +# than its environment has nothing to send. Silence here is a node that enrols, +# writes its files, and then restart-loops on a config error. +warn_missing_db() { + [ "$WARMBLY_ROLE" = "consumer" ] || return 0 + if printf '%s\n' "$NODE_ENV" | grep -q '^PRIMARY_DB='; then + return 0 + fi + warn "" + warn "WARNING: this consumer has no PRIMARY_DB." + warn "" + warn " A consumer is control plane: it updates relational state" + warn " directly, so it needs the database DSN. The control plane sent" + warn " none, which means the backend reads its own DSN from somewhere" + warn " other than its environment (AWS SSM or Secrets Manager)." + warn "" + warn " Add it to $CONFIG_DIR/node.local.env, which re-joining will" + warn " not overwrite, then: systemctl restart warmbly-consumer" + warn "" + warn " PRIMARY_DB=postgres://user:pass@host:5432/warmbly?sslmode=verify-full" + warn "" +} + # render_unit prints the systemd service exactly as install_units writes it. # Separate so it can be asserted on without root, Docker, or a real join: # `make join-check` renders this and checks the result, because every defect @@ -364,7 +421,7 @@ EnvironmentFile=$STATE_DIR/image-ref # any previous one first: a name collision after an unclean stop would # otherwise wedge the service in a restart loop. ExecStartPre=-/usr/bin/docker rm -f $service -ExecStart=/usr/bin/docker run --rm --name $service --env-file $CONFIG_DIR/node.env --network host $MOUNTS \${WARMBLY_IMAGE_REF} +ExecStart=/usr/bin/docker run --rm --name $service --env-file $CONFIG_DIR/node.env --env-file $CONFIG_DIR/node.local.env --network host $MOUNTS \${WARMBLY_IMAGE_REF} ExecStop=/usr/bin/docker stop $service [Install] @@ -473,6 +530,8 @@ start_node() { log "" log " Node id $NODE_ID" log " Version $DESIRED_VERSION" + log " Config $CONFIG_DIR/node.env (rewritten on every join)" + log " Yours $CONFIG_DIR/node.local.env (never rewritten; wins on conflict)" log " Logs journalctl -u $service -f" log " Status systemctl status $service" log "" @@ -504,6 +563,7 @@ main() { install_units start_node warn_shared_blobs + warn_missing_db return 0 } diff --git a/internal/api/routes.go b/internal/api/routes.go index 9472139f..510a6569 100644 --- a/internal/api/routes.go +++ b/internal/api/routes.go @@ -128,6 +128,17 @@ func Run( internal.PUT("/dek/:orgID", h.InternalPutDEK) internal.DELETE("/dek/:orgID", h.InternalDeleteDEK) + // Opens a sealed data key for a node running KMS_PROVIDER=brokered, so + // a machine you own needs no cloud credential of its own. Registered + // before the :orgID routes would ever match it: gin routes the static + // segment first, but keeping them adjacent makes the pair obvious. + internal.POST("/dek/decrypt", h.InternalDecryptDEK) + + // Signs one blob operation for a node running BLOB_PROVIDER=brokered. + // The node then transfers directly against the object store, so bodies + // and attachments never pass through here. + internal.POST("/blobs/presign", h.InternalPresignBlob) + // Click-link tickets: the tracking service resolves /c/ redirects // here instead of touching Postgres (read-only, heavily cached there). internal.GET("/tracked-links/:id", h.InternalGetTrackedLink) diff --git a/internal/app/instancecheck/checks_fleet.go b/internal/app/instancecheck/checks_fleet.go new file mode 100644 index 00000000..beb85866 --- /dev/null +++ b/internal/app/instancecheck/checks_fleet.go @@ -0,0 +1,109 @@ +package instancecheck + +import ( + "context" + "fmt" + "strings" + + "github.com/warmbly/warmbly/internal/config" +) + +const docsSplitDeployment = "/development/split-deployment/" + +func fleetChecks() []check { + return []check{ + {id: "fleet_blobs_not_shared", run: checkFleetBlobsNotShared}, + {id: "fleet_infra_unreachable", run: checkFleetInfraUnreachable}, + } +} + +// fleetSpansMachines reports whether live nodes sit on more than one machine. +// +// Distinct reported addresses, not a node count: replicas on one host all +// report the same address, so `--scale worker=3` on a single-machine install +// stays one machine and the checks below correctly say nothing. It is a +// deliberate under-report — one remote worker and no local node looks like one +// machine — because a false alarm on the default install is worse than a miss +// the join script already warns about. +func fleetSpansMachines(ctx context.Context, d Deps) bool { + if d.DB == nil { + return false + } + var addresses int + err := d.DB.QueryRow(ctx, ` + SELECT count(DISTINCT address) + FROM fleet_nodes + WHERE active + AND address <> '' + AND last_seen_at > now() - $1::interval + `, workerLivenessWindow.String()).Scan(&addresses) + if err != nil { + return false + } + return addresses > 1 +} + +// checkFleetBlobsNotShared catches the failure that looks like everything is +// fine until a send. A worker reads the message body the backend wrote; on +// another machine it has neither the disk nor the permissions, and nothing +// says so until the last step. +func checkFleetBlobsNotShared(ctx context.Context, d Deps, _ Input) *Finding { + provider := config.BlobProvider() + if provider != "filesystem" && provider != "fs" { + return nil + } + if !fleetSpansMachines(ctx, d) { + return nil + } + return result(CategoryData, SeverityError, "Blobs are on local disk and the fleet is not", + "BLOB_PROVIDER is "+provider+", but nodes are checking in from more than one machine. A worker reads the "+ + "message body the backend wrote, so a node that does not share this filesystem cannot send at all, and the "+ + "failure only appears at the last step of a send. Move to BLOB_PROVIDER=s3 with a bucket both sides reach.", + docsSplitDeployment) +} + +// checkFleetInfraUnreachable catches the other silent one: a node is +// configured from the backend's own environment, so an address that only +// resolves here produces a node that enrols cleanly and then reaches nothing. +// It keeps heartbeating over HTTP the whole time, so the fleet looks healthy. +func checkFleetInfraUnreachable(ctx context.Context, d Deps, _ Input) *Finding { + var bad []string + for _, key := range []string{"NATS_URL", "REDIS", "ENCRYPTED_KEYS_BACKEND_URL"} { + v := env(key) + if v == "" { + continue + } + if isLoopbackURL(v) || isContainerInternalURL(v) { + bad = append(bad, fmt.Sprintf("%s=%s", key, v)) + } + } + if len(bad) == 0 { + return nil + } + if !fleetSpansMachines(ctx, d) { + return nil + } + return result(CategoryWorkers, SeverityError, "Nodes are being handed addresses they cannot reach", + "Nodes are checking in from more than one machine, but "+strings.Join(bad, ", ")+ + " resolves only on this host. A joining node inherits these values verbatim, so it enrols, keeps "+ + "heartbeating over HTTP, and silently reaches neither the bus nor the cache. Set them to addresses "+ + "every machine in the fleet can use, then re-join the affected nodes.", + docsSplitDeployment) +} + +// isContainerInternalURL matches the service names the shipped compose file +// uses. They resolve inside that network and nowhere else, and they are what +// an instance that grew out of `make up` is still carrying. +func isContainerInternalURL(raw string) bool { + host := hostOf(raw) + if host == "" { + // A bare host:port (REDIS is sometimes written that way) parses as a + // path rather than a URL, so fall back to the leading label. + host = hostOnly(strings.TrimPrefix(raw, "//")) + } + switch strings.ToLower(host) { + case "nats", "redis", "postgres", "backend", "warmbly-nats", "warmbly-redis": + return true + } + return false +} diff --git a/internal/app/instancecheck/instancecheck.go b/internal/app/instancecheck/instancecheck.go index 39433062..65a86208 100644 --- a/internal/app/instancecheck/instancecheck.go +++ b/internal/app/instancecheck/instancecheck.go @@ -105,6 +105,7 @@ func New(deps Deps) *Registry { r.checks = append(r.checks, mailChecks()...) r.checks = append(r.checks, accessChecks()...) r.checks = append(r.checks, infraChecks()...) + r.checks = append(r.checks, fleetChecks()...) r.checks = append(r.checks, updateChecks()...) return r } diff --git a/internal/app/instanceconfig/entries.go b/internal/app/instanceconfig/entries.go index dc16e669..27b654a9 100644 --- a/internal/app/instanceconfig/entries.go +++ b/internal/app/instanceconfig/entries.go @@ -547,7 +547,7 @@ var table = []Entry{ // Encryption. { Key: "KMS_PROVIDER", Group: GroupEncryption, RuntimeChangeable: ChangeBootOnly, - Effect: "local wraps organization keys with the master key below; aws wraps them with AWS KMS.", + Effect: "local wraps organization keys with the master key below; aws wraps them with AWS KMS; brokered holds no key material and asks this instance to unwrap, which is what a node off this machine should run.", DocsAnchor: docsEncryption, Resolve: func(*Runtime) string { return config.KMSProvider() }, }, @@ -585,7 +585,7 @@ var table = []Entry{ // Storage. { Key: "BLOB_PROVIDER", Group: GroupStorage, RuntimeChangeable: ChangeBootOnly, - Effect: "filesystem stores email bodies, attachments and avatars on disk; s3 stores them in any S3-compatible bucket.", + Effect: "filesystem stores email bodies, attachments and avatars on disk; s3 stores them in any S3-compatible bucket; brokered holds no bucket credential and asks this instance to sign each operation, which is what a node off this machine should run.", DocsAnchor: docsStorage, Resolve: func(*Runtime) string { return config.BlobProvider() }, }, diff --git a/internal/app/worker/wmail/new_email_event_test.go b/internal/app/worker/wmail/new_email_event_test.go index a74fa1f7..4973b434 100644 --- a/internal/app/worker/wmail/new_email_event_test.go +++ b/internal/app/worker/wmail/new_email_event_test.go @@ -7,6 +7,7 @@ import ( "time" "github.com/google/uuid" + "github.com/warmbly/warmbly/internal/infrastructure/storage" "github.com/warmbly/warmbly/internal/models" "github.com/warmbly/warmbly/internal/repository" ) @@ -38,6 +39,9 @@ func (fakeStore) Name() string { return "fake" } func (fakeStore) PresignedGetURL(context.Context, string, time.Duration) (string, error) { return "", nil } +func (fakeStore) PresignedURL(context.Context, storage.PresignOp, string, string, time.Duration) (string, error) { + return "", nil +} type captured struct { eventType models.JobEventType diff --git a/internal/infrastructure/kms/brokered.go b/internal/infrastructure/kms/brokered.go new file mode 100644 index 00000000..014fd911 --- /dev/null +++ b/internal/infrastructure/kms/brokered.go @@ -0,0 +1,122 @@ +package kms + +import ( + "bytes" + "context" + "encoding/base64" + "encoding/json" + "errors" + "fmt" + "net/http" + "net/url" + "strings" + "time" +) + +// BrokeredProvider is the node-side KMS: it holds no key material and no cloud +// credential, and asks the control plane to open a sealed DEK for it. +// +// The point is what a worker then needs to run. With KMS_PROVIDER=aws a node +// must carry an IAM credential good for kms:Decrypt, which means a long-lived +// access key on every machine, in a file `warmbly join` rewrites. Brokering the +// one operation a node actually performs replaces that with the internal API +// token it already has, which is instance-scoped and revocable. +// +// This is not a weaker boundary than the alternative. A node already holds +// CREDENTIALS_ENCRYPTION_KEY and caches plaintext DEKs in Redis: it is trusted +// with plaintext by design, because it is the process that talks to a +// customer's mailbox. What changes is that it is no longer also trusted with a +// credential for the whole account. +// +// The endpoint contract (internal/api/handler/internal_dek.go): +// +// POST {BaseURL}/api/v1/internal/dek/decrypt +// body {"encrypted_data_key":""} +// 200 {"data_key":""} +// +// Auth: Authorization: Bearer +type BrokeredProvider struct { + baseURL string + token string + client *http.Client +} + +// NewBrokered builds the provider. baseURL is the control plane, token is the +// internal API token both sides share. +func NewBrokered(baseURL, token string) (*BrokeredProvider, error) { + if baseURL == "" { + return nil, errors.New("kms.brokered: baseURL is required") + } + if token == "" { + return nil, errors.New("kms.brokered: token is required") + } + if _, err := url.Parse(baseURL); err != nil { + return nil, fmt.Errorf("kms.brokered: invalid baseURL: %w", err) + } + return &BrokeredProvider{ + baseURL: strings.TrimRight(baseURL, "/"), + token: token, + client: &http.Client{Timeout: 15 * time.Second}, + }, nil +} + +func (p *BrokeredProvider) Name() string { return "brokered" } + +// GenerateDataKey is deliberately unavailable. A node only ever handles +// organizations whose key already exists, minted by the control plane when +// their first secret was stored. A node reaching this call means it was asked +// to encrypt for an organization it should never have seen, and creating a key +// there would race the control plane for which one gets stored. +func (p *BrokeredProvider) GenerateDataKey(_ context.Context) ([]byte, string, error) { + return nil, "", errors.New("kms.brokered: a node does not mint data keys; the control plane creates an organization's key when its first secret is stored") +} + +type decryptRequest struct { + EncryptedDataKey string `json:"encrypted_data_key"` +} + +type decryptResponse struct { + DataKey string `json:"data_key"` +} + +func (p *BrokeredProvider) GetDecryptedKey(ctx context.Context, ciphertextB64 string) ([]byte, error) { + if ciphertextB64 == "" { + return nil, errors.New("kms.brokered: empty ciphertext") + } + body, err := json.Marshal(decryptRequest{EncryptedDataKey: ciphertextB64}) + if err != nil { + return nil, err + } + req, err := http.NewRequestWithContext(ctx, http.MethodPost, p.baseURL+"/api/v1/internal/dek/decrypt", bytes.NewReader(body)) + if err != nil { + return nil, err + } + req.Header.Set("Authorization", "Bearer "+p.token) + req.Header.Set("Content-Type", "application/json") + req.Header.Set("User-Agent", "warmbly-node/kms-brokered") + + resp, err := p.client.Do(req) + if err != nil { + return nil, fmt.Errorf("kms.brokered: decrypt: %w", err) + } + defer resp.Body.Close() + + if resp.StatusCode != http.StatusOK { + return nil, fmt.Errorf("kms.brokered: decrypt: unexpected status %d", resp.StatusCode) + } + var out decryptResponse + if err := json.NewDecoder(resp.Body).Decode(&out); err != nil { + return nil, fmt.Errorf("kms.brokered: decode: %w", err) + } + key, err := base64.StdEncoding.DecodeString(out.DataKey) + if err != nil { + return nil, fmt.Errorf("kms.brokered: decode data key: %w", err) + } + // AES-256 everywhere in this system. A short key here means the control + // plane answered with something that is not a DEK, and failing now beats + // sealing data with it. + if len(key) != 32 { + return nil, fmt.Errorf("kms.brokered: data key is %d bytes, want 32", len(key)) + } + return key, nil +} diff --git a/internal/infrastructure/kms/brokered_test.go b/internal/infrastructure/kms/brokered_test.go new file mode 100644 index 00000000..785887d9 --- /dev/null +++ b/internal/infrastructure/kms/brokered_test.go @@ -0,0 +1,100 @@ +package kms + +import ( + "context" + "encoding/base64" + "encoding/json" + "net/http" + "net/http/httptest" + "strings" + "testing" +) + +func TestBrokeredGetDecryptedKey(t *testing.T) { + want := make([]byte, 32) + for i := range want { + want[i] = byte(i) + } + var gotAuth, gotPath, gotCiphertext string + + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + gotAuth = r.Header.Get("Authorization") + gotPath = r.URL.Path + var in decryptRequest + _ = json.NewDecoder(r.Body).Decode(&in) + gotCiphertext = in.EncryptedDataKey + _ = json.NewEncoder(w).Encode(decryptResponse{ + DataKey: base64.StdEncoding.EncodeToString(want), + }) + })) + defer srv.Close() + + p, err := NewBrokered(srv.URL, "tok") + if err != nil { + t.Fatalf("NewBrokered: %v", err) + } + got, err := p.GetDecryptedKey(context.Background(), "sealed") + if err != nil { + t.Fatalf("GetDecryptedKey: %v", err) + } + if string(got) != string(want) { + t.Errorf("key mismatch") + } + if gotAuth != "Bearer tok" { + t.Errorf("auth header %q", gotAuth) + } + if gotPath != "/api/v1/internal/dek/decrypt" { + t.Errorf("path %q", gotPath) + } + if gotCiphertext != "sealed" { + t.Errorf("ciphertext %q", gotCiphertext) + } +} + +// Everything in this system is AES-256. A short key means the control plane +// answered with something that is not a DEK, and sealing data with it would be +// worse than failing. +func TestBrokeredRejectsShortKey(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + _ = json.NewEncoder(w).Encode(decryptResponse{ + DataKey: base64.StdEncoding.EncodeToString([]byte("too short")), + }) + })) + defer srv.Close() + + p, _ := NewBrokered(srv.URL, "tok") + if _, err := p.GetDecryptedKey(context.Background(), "sealed"); err == nil { + t.Fatal("a 9-byte data key was accepted") + } +} + +// A node must never mint an organization's key: the control plane does that +// when the org's first secret is stored, and a second minter races it. +func TestBrokeredCannotGenerate(t *testing.T) { + p, _ := NewBrokered("https://x", "tok") + if _, _, err := p.GenerateDataKey(context.Background()); err == nil { + t.Fatal("a node was allowed to mint a data key") + } +} + +func TestBrokeredErrorStatus(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(http.StatusUnauthorized) + })) + defer srv.Close() + + p, _ := NewBrokered(srv.URL, "tok") + _, err := p.GetDecryptedKey(context.Background(), "sealed") + if err == nil || !strings.Contains(err.Error(), "401") { + t.Fatalf("got %v, want an error naming the status", err) + } +} + +func TestNewBrokeredRequiresBaseURLAndToken(t *testing.T) { + if _, err := NewBrokered("", "tok"); err == nil { + t.Error("empty baseURL accepted") + } + if _, err := NewBrokered("https://x", ""); err == nil { + t.Error("empty token accepted") + } +} diff --git a/internal/infrastructure/kms/factory.go b/internal/infrastructure/kms/factory.go index 73deb8a3..1ea92a79 100644 --- a/internal/infrastructure/kms/factory.go +++ b/internal/infrastructure/kms/factory.go @@ -10,9 +10,11 @@ import ( // FromEnv constructs the active KMS provider from environment variables. // -// KMS_PROVIDER=aws -> NewAWS (uses awscfg, KMS_AWS_KEY_ID or fallbackAWSKeyID) -// KMS_PROVIDER=local -> NewLocalFromEnv -// (unset) -> defaults to "aws" for backwards compatibility +// KMS_PROVIDER=aws -> NewAWS (uses awscfg, KMS_AWS_KEY_ID or fallbackAWSKeyID) +// KMS_PROVIDER=local -> NewLocalFromEnv +// KMS_PROVIDER=brokered -> node-side: no key material, no cloud credential; +// the control plane opens sealed keys for it +// (unset) -> defaults to "aws" for backwards compatibility // // Callers pass an AWS config that's only consulted when the AWS provider is // selected. The fallbackAWSKeyID is used when KMS_AWS_KEY_ID is empty — this @@ -34,7 +36,18 @@ func FromEnv(ctx context.Context, awscfg aws.Config, fallbackAWSKeyID string) (P return New(ctx, awscfg, keyID) case "local": return NewLocalFromEnv() + case "brokered": + // Same pair every other internal-API client on a node uses. + baseURL := os.Getenv("ENCRYPTED_KEYS_BACKEND_URL") + if baseURL == "" { + baseURL = os.Getenv("WARMBLY_BACKEND_URL") + } + token := os.Getenv("INTERNAL_API_TOKEN") + if token == "" { + token = os.Getenv("ENCRYPTED_KEYS_WORKER_TOKEN") + } + return NewBrokered(baseURL, token) default: - return nil, fmt.Errorf("kms: unknown KMS_PROVIDER %q (want: aws, local)", provider) + return nil, fmt.Errorf("kms: unknown KMS_PROVIDER %q (want: aws, local, brokered)", provider) } } diff --git a/internal/infrastructure/kms/provider.go b/internal/infrastructure/kms/provider.go index 5f74671d..75ce22d1 100644 --- a/internal/infrastructure/kms/provider.go +++ b/internal/infrastructure/kms/provider.go @@ -29,6 +29,7 @@ type Provider interface { var ( _ Provider = (*KMS)(nil) _ Provider = (*LocalProvider)(nil) + _ Provider = (*BrokeredProvider)(nil) ) // Name satisfies Provider for the AWS implementation. diff --git a/internal/infrastructure/storage/brokered.go b/internal/infrastructure/storage/brokered.go new file mode 100644 index 00000000..5511801d --- /dev/null +++ b/internal/infrastructure/storage/brokered.go @@ -0,0 +1,288 @@ +package storage + +import ( + "bytes" + "context" + "encoding/json" + "errors" + "fmt" + "io" + "net/http" + "net/url" + "strings" + "time" +) + +// BrokeredStore is the node-side blob backend: it holds no credential for the +// object store and asks the control plane to sign each operation instead, then +// sends the bytes straight to the store. +// +// This is what lets a worker on a machine you own carry no cloud credentials. +// The alternative is copying a long-lived access key onto every box, into a +// file `warmbly join` rewrites, where a compromised worker yields the whole +// bucket. A signed URL is one verb, one key, and expires. +// +// Bytes never pass through the backend, so the control plane pays no bandwidth +// for a mailbox sync and stays out of the data path. +// +// The endpoint contract (internal/api/handler/internal_blobs.go): +// +// POST {BaseURL}/api/v1/internal/blobs/presign +// body {"op":"get|put|head|delete","key":"...","content_type":"..."} +// 200 {"url":"...","method":"GET","expires_in":300} +// 501 the instance's blob backend cannot sign (filesystem) +// +// Auth: Authorization: Bearer +type BrokeredStore struct { + baseURL string + token string + client *http.Client +} + +// maxBrokeredBody caps what Put will buffer when the caller's reader cannot +// report its length. S3 rejects a chunked upload against a presigned URL, so +// the length has to be known before the request starts. Callers in this repo +// hand over a *bytes.Reader and never reach the buffer at all. +const maxBrokeredBody = 64 << 20 + +// brokerTTL is how long a signed URL stays valid. Long enough for a slow +// transfer on a bad link, short enough that a URL in a log is stale by the +// time anyone reads it. +const brokerTTL = 5 * time.Minute + +// BrokeredOption configures a BrokeredStore. +type BrokeredOption func(*BrokeredStore) + +// WithBrokeredHTTPClient overrides the client used for both the broker call +// and the transfer itself. +func WithBrokeredHTTPClient(c *http.Client) BrokeredOption { + return func(s *BrokeredStore) { s.client = c } +} + +func NewBrokered(baseURL, token string, opts ...BrokeredOption) (*BrokeredStore, error) { + if baseURL == "" { + return nil, errors.New("storage.brokered: baseURL is required") + } + if token == "" { + return nil, errors.New("storage.brokered: token is required") + } + if _, err := url.Parse(baseURL); err != nil { + return nil, fmt.Errorf("storage.brokered: invalid baseURL: %w", err) + } + s := &BrokeredStore{ + baseURL: strings.TrimRight(baseURL, "/"), + token: token, + // No global timeout: a large attachment on a slow link is a legitimate + // long request, and the per-call context already bounds it. + client: &http.Client{}, + } + for _, o := range opts { + o(s) + } + return s, nil +} + +func (s *BrokeredStore) Name() string { return "brokered" } + +type presignRequest struct { + Op string `json:"op"` + Key string `json:"key"` + ContentType string `json:"content_type,omitempty"` +} + +type presignResponse struct { + URL string `json:"url"` + Method string `json:"method"` +} + +// presign asks the control plane for a URL. Every operation starts here, so +// this is also where a revoked token or a backend that cannot sign surfaces, +// with the reason rather than a bare failure at transfer time. +func (s *BrokeredStore) presign(ctx context.Context, op PresignOp, key, contentType string) (presignResponse, error) { + var out presignResponse + if key == "" { + return out, errors.New("storage.brokered: key is required") + } + body, err := json.Marshal(presignRequest{Op: string(op), Key: key, ContentType: contentType}) + if err != nil { + return out, err + } + req, err := http.NewRequestWithContext(ctx, http.MethodPost, s.baseURL+"/api/v1/internal/blobs/presign", bytes.NewReader(body)) + if err != nil { + return out, err + } + req.Header.Set("Authorization", "Bearer "+s.token) + req.Header.Set("Content-Type", "application/json") + req.Header.Set("User-Agent", "warmbly-node/storage-brokered") + + resp, err := s.client.Do(req) + if err != nil { + return out, fmt.Errorf("storage.brokered: presign %s: %w", op, err) + } + defer resp.Body.Close() + + switch resp.StatusCode { + case http.StatusOK: + case http.StatusNotImplemented: + return out, fmt.Errorf("storage.brokered: %w: the instance's blob backend cannot sign URLs (BLOB_PROVIDER=filesystem)", ErrUnsupported) + default: + return out, fmt.Errorf("storage.brokered: presign %s: unexpected status %d", op, resp.StatusCode) + } + if err := json.NewDecoder(resp.Body).Decode(&out); err != nil { + return out, fmt.Errorf("storage.brokered: decode presign: %w", err) + } + if out.URL == "" { + return out, errors.New("storage.brokered: control plane returned no url") + } + return out, nil +} + +// do runs one signed request. The response body is handed back open for Get +// and closed by every other caller. +func (s *BrokeredStore) do(ctx context.Context, signed presignResponse, body io.Reader, length int64, contentType string) (*http.Response, error) { + req, err := http.NewRequestWithContext(ctx, signed.Method, signed.URL, body) + if err != nil { + return nil, err + } + if length >= 0 { + req.ContentLength = length + } + // The signature covers this header when it was signed with one, so it has + // to go back exactly as asked for. + if contentType != "" { + req.Header.Set("Content-Type", contentType) + } + return s.client.Do(req) +} + +func (s *BrokeredStore) Get(ctx context.Context, key string) (io.ReadCloser, error) { + signed, err := s.presign(ctx, PresignOpGet, key, "") + if err != nil { + return nil, err + } + resp, err := s.do(ctx, signed, nil, -1, "") + if err != nil { + return nil, fmt.Errorf("storage.brokered: get: %w", err) + } + switch resp.StatusCode { + case http.StatusOK, http.StatusPartialContent: + return resp.Body, nil + case http.StatusNotFound, http.StatusForbidden: + // A bucket with ListBucket withheld answers a missing key with 403 + // rather than 404, and a caller checking for ErrNotFound must not have + // to know which of the two it is talking to. + resp.Body.Close() + return nil, ErrNotFound + default: + resp.Body.Close() + return nil, fmt.Errorf("storage.brokered: get: unexpected status %d", resp.StatusCode) + } +} + +// knownLength reports the size of the readers this repo actually passes to +// Put, so the common path streams without a copy. +func knownLength(r io.Reader) (int64, bool) { + switch v := r.(type) { + case *bytes.Reader: + return int64(v.Len()), true + case *bytes.Buffer: + return int64(v.Len()), true + case *strings.Reader: + return int64(v.Len()), true + } + return 0, false +} + +func (s *BrokeredStore) Put(ctx context.Context, key string, body io.Reader, contentType string) error { + length, ok := knownLength(body) + if !ok { + // A presigned PUT is signed for a plain request, and S3 rejects the + // chunked encoding Go would otherwise choose for an unknown length. + buf, err := io.ReadAll(io.LimitReader(body, maxBrokeredBody+1)) + if err != nil { + return fmt.Errorf("storage.brokered: put: %w", err) + } + if len(buf) > maxBrokeredBody { + return fmt.Errorf("storage.brokered: put: object exceeds %d bytes", maxBrokeredBody) + } + body = bytes.NewReader(buf) + length = int64(len(buf)) + } + + signed, err := s.presign(ctx, PresignOpPut, key, contentType) + if err != nil { + return err + } + resp, err := s.do(ctx, signed, body, length, contentType) + if err != nil { + return fmt.Errorf("storage.brokered: put: %w", err) + } + defer resp.Body.Close() + if resp.StatusCode != http.StatusOK && resp.StatusCode != http.StatusCreated { + return fmt.Errorf("storage.brokered: put: unexpected status %d", resp.StatusCode) + } + return nil +} + +func (s *BrokeredStore) Has(ctx context.Context, key string) (bool, error) { + signed, err := s.presign(ctx, PresignOpHead, key, "") + if err != nil { + return false, err + } + resp, err := s.do(ctx, signed, nil, -1, "") + if err != nil { + return false, fmt.Errorf("storage.brokered: has: %w", err) + } + defer resp.Body.Close() + switch resp.StatusCode { + case http.StatusOK: + return true, nil + case http.StatusNotFound, http.StatusForbidden: + return false, nil + default: + return false, fmt.Errorf("storage.brokered: has: unexpected status %d", resp.StatusCode) + } +} + +func (s *BrokeredStore) Delete(ctx context.Context, key string) error { + signed, err := s.presign(ctx, PresignOpDelete, key, "") + if err != nil { + return err + } + resp, err := s.do(ctx, signed, nil, -1, "") + if err != nil { + return fmt.Errorf("storage.brokered: delete: %w", err) + } + defer resp.Body.Close() + // S3 answers a delete of a missing key with 204, and the Store contract + // says that is not an error either way. + if resp.StatusCode != http.StatusNoContent && resp.StatusCode != http.StatusOK && resp.StatusCode != http.StatusNotFound { + return fmt.Errorf("storage.brokered: delete: unexpected status %d", resp.StatusCode) + } + return nil +} + +// PutPublic is not available to a node. Public objects are avatars and org +// logos, which only the backend writes, and serving them needs a public base +// URL a node has no way to know. +func (s *BrokeredStore) PutPublic(_ context.Context, _ string, _ io.Reader, _ string) (string, error) { + return "", ErrUnsupported +} + +func (s *BrokeredStore) PresignedGetURL(ctx context.Context, key string, _ time.Duration) (string, error) { + return s.PresignedURL(ctx, PresignOpGet, key, "", 0) +} + +// PresignedURL passes the request through to the control plane. The ttl is the +// broker's to choose: a node asking for a longer-lived URL than the instance +// wants to issue is exactly what the broker exists to refuse. +func (s *BrokeredStore) PresignedURL(ctx context.Context, op PresignOp, key, contentType string, _ time.Duration) (string, error) { + if !op.Valid() { + return "", fmt.Errorf("storage.brokered: unknown op %q", op) + } + signed, err := s.presign(ctx, op, key, contentType) + if err != nil { + return "", err + } + return signed.URL, nil +} diff --git a/internal/infrastructure/storage/brokered_test.go b/internal/infrastructure/storage/brokered_test.go new file mode 100644 index 00000000..224b8b52 --- /dev/null +++ b/internal/infrastructure/storage/brokered_test.go @@ -0,0 +1,209 @@ +package storage + +import ( + "bytes" + "context" + "encoding/json" + "errors" + "io" + "net/http" + "net/http/httptest" + "strings" + "testing" +) + +// brokerFor stands up a fake control plane and a fake object store, wired the +// way the real pair is: the broker hands back a URL into the store, and the +// node is expected to talk to the store and not to the broker. +func brokerFor(t *testing.T, store http.Handler) (*BrokeredStore, *int) { + t.Helper() + objects := httptest.NewServer(store) + t.Cleanup(objects.Close) + + presignCalls := 0 + broker := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Header.Get("Authorization") != "Bearer tok" { + w.WriteHeader(http.StatusUnauthorized) + return + } + var req presignRequest + if err := json.NewDecoder(r.Body).Decode(&req); err != nil { + w.WriteHeader(http.StatusBadRequest) + return + } + presignCalls++ + op := PresignOp(req.Op) + if !op.Valid() { + w.WriteHeader(http.StatusBadRequest) + return + } + _ = json.NewEncoder(w).Encode(presignResponse{ + URL: objects.URL + "/" + req.Key, + Method: op.Method(), + }) + })) + t.Cleanup(broker.Close) + + s, err := NewBrokered(broker.URL, "tok") + if err != nil { + t.Fatalf("NewBrokered: %v", err) + } + return s, &presignCalls +} + +func TestBrokeredGet(t *testing.T) { + s, calls := brokerFor(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Method != http.MethodGet { + t.Errorf("object store saw %s, want GET", r.Method) + } + _, _ = io.WriteString(w, "body bytes") + })) + + rc, err := s.Get(context.Background(), "emails/1") + if err != nil { + t.Fatalf("Get: %v", err) + } + defer rc.Close() + got, _ := io.ReadAll(rc) + if string(got) != "body bytes" { + t.Errorf("got %q, want %q", got, "body bytes") + } + if *calls != 1 { + t.Errorf("presign called %d times, want 1", *calls) + } +} + +// A bucket that withholds ListBucket answers a missing key with 403 rather +// than 404, and callers check for ErrNotFound without knowing which. +func TestBrokeredGetMissingIsNotFound(t *testing.T) { + for _, status := range []int{http.StatusNotFound, http.StatusForbidden} { + s, _ := brokerFor(t, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(status) + })) + if _, err := s.Get(context.Background(), "gone"); !errors.Is(err, ErrNotFound) { + t.Errorf("status %d: got %v, want ErrNotFound", status, err) + } + } +} + +// A presigned PUT is signed for a plain request; S3 rejects the chunked +// encoding Go picks when it cannot tell the length. The reader this repo +// passes is a *bytes.Reader, so the length must survive to the request. +func TestBrokeredPutSetsContentLength(t *testing.T) { + var gotLen int64 = -1 + var gotType string + var gotBody []byte + s, _ := brokerFor(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + gotLen = r.ContentLength + gotType = r.Header.Get("Content-Type") + gotBody, _ = io.ReadAll(r.Body) + if r.Method != http.MethodPut { + t.Errorf("object store saw %s, want PUT", r.Method) + } + w.WriteHeader(http.StatusOK) + })) + + if err := s.Put(context.Background(), "k", bytes.NewReader([]byte("hello")), "text/plain"); err != nil { + t.Fatalf("Put: %v", err) + } + if gotLen != 5 { + t.Errorf("Content-Length %d, want 5", gotLen) + } + // The signature covers the content type when one was signed, so it has to + // go back exactly as asked for. + if gotType != "text/plain" { + t.Errorf("Content-Type %q, want text/plain", gotType) + } + if string(gotBody) != "hello" { + t.Errorf("body %q, want hello", gotBody) + } +} + +// A reader with no length still has to produce a length, by buffering. +func TestBrokeredPutUnknownLengthIsBuffered(t *testing.T) { + var gotLen int64 = -1 + s, _ := brokerFor(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + gotLen = r.ContentLength + _, _ = io.Copy(io.Discard, r.Body) + w.WriteHeader(http.StatusOK) + })) + + // io.MultiReader reports no length, unlike the concrete reader types. + body := io.MultiReader(strings.NewReader("abc"), strings.NewReader("de")) + if err := s.Put(context.Background(), "k", body, ""); err != nil { + t.Fatalf("Put: %v", err) + } + if gotLen != 5 { + t.Errorf("Content-Length %d, want 5", gotLen) + } +} + +func TestBrokeredHasAndDelete(t *testing.T) { + var methods []string + s, _ := brokerFor(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + methods = append(methods, r.Method) + if r.Method == http.MethodHead { + w.WriteHeader(http.StatusOK) + return + } + w.WriteHeader(http.StatusNoContent) + })) + + ok, err := s.Has(context.Background(), "k") + if err != nil || !ok { + t.Fatalf("Has = %v, %v; want true, nil", ok, err) + } + if err := s.Delete(context.Background(), "k"); err != nil { + t.Fatalf("Delete: %v", err) + } + if len(methods) != 2 || methods[0] != http.MethodHead || methods[1] != http.MethodDelete { + t.Errorf("object store saw %v, want [HEAD DELETE]", methods) + } +} + +// The filesystem backend cannot sign, and the broker says so with 501. That +// has to reach the caller as ErrUnsupported rather than a bare failure, since +// the fix is a different BLOB_PROVIDER and not a retry. +func TestBrokeredUnsupportedBackend(t *testing.T) { + broker := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(http.StatusNotImplemented) + })) + defer broker.Close() + + s, err := NewBrokered(broker.URL, "tok") + if err != nil { + t.Fatalf("NewBrokered: %v", err) + } + if _, err := s.Get(context.Background(), "k"); !errors.Is(err, ErrUnsupported) { + t.Errorf("got %v, want ErrUnsupported", err) + } +} + +func TestNewBrokeredRequiresBaseURLAndToken(t *testing.T) { + if _, err := NewBrokered("", "tok"); err == nil { + t.Error("empty baseURL accepted") + } + if _, err := NewBrokered("https://x", ""); err == nil { + t.Error("empty token accepted") + } +} + +func TestPresignOpMethods(t *testing.T) { + cases := map[PresignOp]string{ + PresignOpGet: http.MethodGet, + PresignOpPut: http.MethodPut, + PresignOpHead: http.MethodHead, + PresignOpDelete: http.MethodDelete, + } + for op, want := range cases { + if !op.Valid() { + t.Errorf("%s reported invalid", op) + } + if got := op.Method(); got != want { + t.Errorf("%s method = %s, want %s", op, got, want) + } + } + if PresignOp("list").Valid() { + t.Error("unknown op reported valid") + } +} diff --git a/internal/infrastructure/storage/factory.go b/internal/infrastructure/storage/factory.go index 0144c2c0..a94bd20b 100644 --- a/internal/infrastructure/storage/factory.go +++ b/internal/infrastructure/storage/factory.go @@ -13,6 +13,8 @@ import ( // BLOB_PROVIDER=s3 -> existing S3 client (works for AWS / MinIO / // R2 / B2 / Hetzner Object Storage) // BLOB_PROVIDER=filesystem -> NewFilesystem at BLOB_FS_ROOT +// BLOB_PROVIDER=brokered -> node-side: no bucket credential, every +// operation signed by the control plane // (unset) -> defaults to "s3" for backwards compatibility // // awscfg + defaultBucket are only consulted when the S3 provider is selected. @@ -51,8 +53,21 @@ func NewFromEnv(ctx context.Context, awscfg aws.Config, defaultBucket string) (S return nil, fmt.Errorf("storage: filesystem provider requires BLOB_FS_ROOT") } return NewFilesystem(root, publicBaseURL) + case "brokered": + // Same pair the node's other internal-API clients use, so a node has + // exactly one credential for the control plane and none for anything + // behind it. + baseURL := os.Getenv("ENCRYPTED_KEYS_BACKEND_URL") + if baseURL == "" { + baseURL = os.Getenv("WARMBLY_BACKEND_URL") + } + token := os.Getenv("INTERNAL_API_TOKEN") + if token == "" { + token = os.Getenv("ENCRYPTED_KEYS_WORKER_TOKEN") + } + return NewBrokered(baseURL, token) default: - return nil, fmt.Errorf("storage: unknown BLOB_PROVIDER %q (want: s3, filesystem)", provider) + return nil, fmt.Errorf("storage: unknown BLOB_PROVIDER %q (want: s3, filesystem, brokered)", provider) } } @@ -60,4 +75,5 @@ func NewFromEnv(ctx context.Context, awscfg aws.Config, defaultBucket string) (S var ( _ Store = (*Client)(nil) _ Store = (*FilesystemStore)(nil) + _ Store = (*BrokeredStore)(nil) ) diff --git a/internal/infrastructure/storage/filesystem.go b/internal/infrastructure/storage/filesystem.go index a0bcb9ad..87653187 100644 --- a/internal/infrastructure/storage/filesystem.go +++ b/internal/infrastructure/storage/filesystem.go @@ -150,3 +150,10 @@ func (s *FilesystemStore) Has(_ context.Context, key string) (bool, error) { func (s *FilesystemStore) PresignedGetURL(_ context.Context, _ string, _ time.Duration) (string, error) { return "", ErrUnsupported } + +// PresignedURL is unsupported for the same reason, for every verb. A node on +// another machine therefore cannot reach filesystem blobs at all, which is the +// honest answer: they are on a disk it does not have. +func (s *FilesystemStore) PresignedURL(_ context.Context, _ PresignOp, _, _ string, _ time.Duration) (string, error) { + return "", ErrUnsupported +} diff --git a/internal/infrastructure/storage/s3_store.go b/internal/infrastructure/storage/s3_store.go index 91dc1352..398a209a 100644 --- a/internal/infrastructure/storage/s3_store.go +++ b/internal/infrastructure/storage/s3_store.go @@ -86,13 +86,68 @@ func (c *Client) Has(ctx context.Context, key string) (bool, error) { } func (c *Client) PresignedGetURL(ctx context.Context, key string, ttl time.Duration) (string, error) { + return c.PresignedURL(ctx, PresignOpGet, key, "", ttl) +} + +// PresignedURL signs one operation on one key. This is what lets a worker hold +// no bucket credentials at all: the control plane signs, the node sends the +// bytes straight to the store, and nothing is proxied. +// +// A signature covers the verb, so a URL minted for a read cannot be used to +// overwrite. +func (c *Client) PresignedURL(ctx context.Context, op PresignOp, key, contentType string, ttl time.Duration) (string, error) { ps := s3.NewPresignClient(c.Client) - out, err := ps.PresignGetObject(ctx, &s3.GetObjectInput{ - Bucket: aws.String(c.Bucket), - Key: aws.String(key), - }, s3.WithPresignExpires(ttl)) - if err != nil { - return "", err + expires := s3.WithPresignExpires(ttl) + + switch op { + case PresignOpGet: + out, err := ps.PresignGetObject(ctx, &s3.GetObjectInput{ + Bucket: aws.String(c.Bucket), + Key: aws.String(key), + }, expires) + if err != nil { + return "", err + } + return out.URL, nil + + case PresignOpPut: + in := &s3.PutObjectInput{ + Bucket: aws.String(c.Bucket), + Key: aws.String(key), + } + // Signed when present, so the sender must send the same header back. + // Left unsigned when empty rather than defaulted, which would make + // every unspecified upload fail the signature check. + if contentType != "" { + in.ContentType = aws.String(contentType) + } + out, err := ps.PresignPutObject(ctx, in, expires) + if err != nil { + return "", err + } + return out.URL, nil + + case PresignOpHead: + out, err := ps.PresignHeadObject(ctx, &s3.HeadObjectInput{ + Bucket: aws.String(c.Bucket), + Key: aws.String(key), + }, expires) + if err != nil { + return "", err + } + return out.URL, nil + + case PresignOpDelete: + out, err := ps.PresignDeleteObject(ctx, &s3.DeleteObjectInput{ + Bucket: aws.String(c.Bucket), + Key: aws.String(key), + }, expires) + if err != nil { + return "", err + } + return out.URL, nil + + default: + return "", fmt.Errorf("storage: cannot presign unknown op %q", op) } - return out.URL, nil } diff --git a/internal/infrastructure/storage/store.go b/internal/infrastructure/storage/store.go index 1f1cf924..a393b845 100644 --- a/internal/infrastructure/storage/store.go +++ b/internal/infrastructure/storage/store.go @@ -3,9 +3,46 @@ package storage import ( "context" "io" + "net/http" "time" ) +// PresignOp is the verb a presigned URL authorises. The set is what a node +// running against object storage it has no credentials for actually needs: +// read a body, write one, check existence, remove it. +type PresignOp string + +const ( + PresignOpGet PresignOp = "get" + PresignOpPut PresignOp = "put" + PresignOpHead PresignOp = "head" + PresignOpDelete PresignOp = "delete" +) + +// Valid reports whether op is one this package knows how to sign. Callers that +// take an op off the wire must check it before use. +func (o PresignOp) Valid() bool { + switch o { + case PresignOpGet, PresignOpPut, PresignOpHead, PresignOpDelete: + return true + } + return false +} + +// Method is the HTTP verb a caller sends to a URL signed for op. +func (o PresignOp) Method() string { + switch o { + case PresignOpPut: + return http.MethodPut + case PresignOpHead: + return http.MethodHead + case PresignOpDelete: + return http.MethodDelete + default: + return http.MethodGet + } +} + // Store is the abstraction over blob storage. One implementation suffices for // AWS S3, MinIO, Cloudflare R2, Backblaze B2, and Hetzner Object Storage — all // speak the S3 protocol. A separate Filesystem implementation covers @@ -41,6 +78,13 @@ type Store interface { // produce one (filesystem, in particular) should return ErrUnsupported. PresignedGetURL(ctx context.Context, key string, ttl time.Duration) (string, error) + // PresignedURL is PresignedGetURL generalised to the other verbs, so a + // caller that holds no credential for the bucket can still work against it + // through a party that does. contentType applies to PresignOpPut and is + // ignored otherwise. Same contract on backends that cannot sign: + // ErrUnsupported. + PresignedURL(ctx context.Context, op PresignOp, key, contentType string, ttl time.Duration) (string, error) + // Name returns the implementation identifier for admin UI / audit logs. Name() string } diff --git a/scripts/aws-bootstrap.sh b/scripts/aws-bootstrap.sh new file mode 100644 index 00000000..0ee6a55c --- /dev/null +++ b/scripts/aws-bootstrap.sh @@ -0,0 +1,321 @@ +#!/bin/sh +# Create the AWS side of a split deployment: a KMS key, a bucket, a database, +# and the two IAM users that reach them. +# +# scripts/aws-bootstrap.sh --domain example.com --region eu-central-1 +# +# Idempotent: every step checks for what it would create and reports it rather +# than failing, so a re-run after fixing one answer is safe. +# +# The part worth reading is the two policies. A control-plane credential mints +# and opens data keys and reads and writes the bucket; a node credential should +# be able to do neither, and it is the one that ends up on machines in a +# datacentre you share. Warmbly's brokered providers mean a node normally needs +# no AWS credential at all (KMS_PROVIDER=brokered, BLOB_PROVIDER=brokered), so +# --with-node-user is off by default and exists for the deliberate exception. +set -eu + +REGION="" +DOMAIN="" +PREFIX="warmbly" +DB_INSTANCE_CLASS="db.t4g.micro" +DB_STORAGE_GB="20" +WITH_NODE_USER="false" +SKIP_DB="false" +DRY_RUN="false" + +log() { printf '%s\n' "$*"; } +warn() { printf '%s\n' "$*" >&2; } +die() { printf 'error: %s\n' "$*" >&2; exit 1; } + +usage() { + cat <<'USAGE' +Create the AWS resources a split Warmbly deployment needs. + + --domain Your domain, used to verify an SES identity. (required) + --region AWS region. Put it next to your container host. (required) + --prefix

Name prefix for every resource. Default warmbly. + --db-class RDS instance class. Default db.t4g.micro. + --db-storage RDS allocated storage in GB. Default 20. + --skip-db Create everything except the database. + --with-node-user Also create an IAM user for nodes that reach AWS + directly. Not needed with the brokered providers. + --dry-run Print what would be created, change nothing. + -h, --help This text. + +Needs the AWS CLI, authenticated as someone who can create KMS keys, S3 +buckets, RDS instances and IAM users. +USAGE +} + +parse_args() { + while [ $# -gt 0 ]; do + case "$1" in + --domain) DOMAIN="${2:-}"; shift 2 ;; + --region) REGION="${2:-}"; shift 2 ;; + --prefix) PREFIX="${2:-}"; shift 2 ;; + --db-class) DB_INSTANCE_CLASS="${2:-}"; shift 2 ;; + --db-storage) DB_STORAGE_GB="${2:-}"; shift 2 ;; + --skip-db) SKIP_DB="true"; shift ;; + --with-node-user) WITH_NODE_USER="true"; shift ;; + --dry-run) DRY_RUN="true"; shift ;; + -h|--help) usage; exit 0 ;; + *) die "unknown option: $1 (try --help)" ;; + esac + done +} + +require_args() { + [ -n "$DOMAIN" ] || die "--domain is required" + [ -n "$REGION" ] || die "--region is required" + command -v aws >/dev/null 2>&1 || die "the aws CLI is required but not installed" + ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text) \ + || die "could not read the current AWS identity; is the CLI authenticated?" + log "Account $ACCOUNT_ID, region $REGION" +} + +run() { + if [ "$DRY_RUN" = "true" ]; then + log " would run: $*" + return 0 + fi + "$@" +} + +# ---- KMS ------------------------------------------------------------------- + +create_kms() { + KEY_ALIAS="alias/$PREFIX" + if aws kms describe-key --key-id "$KEY_ALIAS" --region "$REGION" >/dev/null 2>&1; then + log "KMS: $KEY_ALIAS already exists" + return 0 + fi + log "KMS: creating $KEY_ALIAS" + if [ "$DRY_RUN" = "true" ]; then + log " would create a symmetric key and alias it $KEY_ALIAS" + return 0 + fi + key_id=$(aws kms create-key \ + --description "Warmbly per-organization data keys" \ + --region "$REGION" \ + --query 'KeyMetadata.KeyId' --output text) + aws kms create-alias --alias-name "$KEY_ALIAS" --target-key-id "$key_id" --region "$REGION" + # Losing this key makes every stored mailbox credential unreadable, so it + # gets the longest window AWS offers against an accidental delete. + aws kms enable-key-rotation --key-id "$key_id" --region "$REGION" || true + log "KMS: created $KEY_ALIAS ($key_id)" +} + +# ---- S3 -------------------------------------------------------------------- + +create_bucket() { + BUCKET="$PREFIX-blobs-$ACCOUNT_ID" + if aws s3api head-bucket --bucket "$BUCKET" >/dev/null 2>&1; then + log "S3: $BUCKET already exists" + return 0 + fi + log "S3: creating $BUCKET" + if [ "$DRY_RUN" = "true" ]; then + log " would create $BUCKET with public access blocked and SSE enabled" + return 0 + fi + if [ "$REGION" = "us-east-1" ]; then + aws s3api create-bucket --bucket "$BUCKET" --region "$REGION" + else + aws s3api create-bucket --bucket "$BUCKET" --region "$REGION" \ + --create-bucket-configuration "LocationConstraint=$REGION" + fi + aws s3api put-public-access-block --bucket "$BUCKET" \ + --public-access-block-configuration \ + "BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true" + aws s3api put-bucket-encryption --bucket "$BUCKET" \ + --server-side-encryption-configuration \ + '{"Rules":[{"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm":"AES256"}}]}' + log "S3: created $BUCKET" +} + +# ---- IAM ------------------------------------------------------------------- + +# control_policy is what the backend and consumer need: mint and open data +# keys, read and write the bucket, send platform mail. +control_policy() { + cat </dev/null 2>&1; then + log "IAM: $user already exists (policy updated, no new access key)" + else + log "IAM: creating $user" + run aws iam create-user --user-name "$user" + fi + if [ "$DRY_RUN" = "true" ]; then + log " would attach an inline policy to $user" + return 0 + fi + printf '%s' "$policy_json" > "/tmp/$user-policy.json" + aws iam put-user-policy --user-name "$user" \ + --policy-name "$user-policy" --policy-document "file:///tmp/$user-policy.json" + rm -f "/tmp/$user-policy.json" +} + +create_users() { + create_user "$PREFIX-control" "$(control_policy)" + if [ "$WITH_NODE_USER" = "true" ]; then + create_user "$PREFIX-node" "$(node_policy)" + else + log "IAM: skipping the node user; nodes use the brokered providers and need no AWS credential" + fi +} + +# ---- RDS ------------------------------------------------------------------- + +create_db() { + if [ "$SKIP_DB" = "true" ]; then + log "RDS: skipped" + return 0 + fi + DB_ID="$PREFIX-db" + if aws rds describe-db-instances --db-instance-identifier "$DB_ID" --region "$REGION" >/dev/null 2>&1; then + log "RDS: $DB_ID already exists" + return 0 + fi + log "RDS: creating $DB_ID ($DB_INSTANCE_CLASS, ${DB_STORAGE_GB}GB)" + if [ "$DRY_RUN" = "true" ]; then + log " would create a publicly accessible Postgres 16 instance with storage encrypted" + return 0 + fi + DB_PASSWORD=$(openssl rand -base64 30 | tr -d '/+=' | cut -c1-28) + aws rds create-db-instance \ + --db-instance-identifier "$DB_ID" \ + --db-instance-class "$DB_INSTANCE_CLASS" \ + --engine postgres \ + --engine-version 16 \ + --allocated-storage "$DB_STORAGE_GB" \ + --storage-type gp3 \ + --storage-encrypted \ + --master-username warmbly \ + --master-user-password "$DB_PASSWORD" \ + --db-name warmbly \ + --backup-retention-period 7 \ + --publicly-accessible \ + --no-multi-az \ + --region "$REGION" >/dev/null + log "RDS: creating. The master password is printed once, below." + DB_PASSWORD_PRINTED="$DB_PASSWORD" +} + +# ---- SES ------------------------------------------------------------------- + +create_ses_identity() { + if aws sesv2 get-email-identity --email-identity "$DOMAIN" --region "$REGION" >/dev/null 2>&1; then + log "SES: $DOMAIN is already an identity" + return 0 + fi + log "SES: creating a domain identity for $DOMAIN" + if [ "$DRY_RUN" = "true" ]; then + log " would create an SES domain identity with Easy DKIM" + return 0 + fi + aws sesv2 create-email-identity --email-identity "$DOMAIN" --region "$REGION" >/dev/null + log "SES: publish the DKIM records it now expects:" + aws sesv2 get-email-identity --email-identity "$DOMAIN" --region "$REGION" \ + --query 'DkimAttributes.Tokens' --output text 2>/dev/null || true +} + +summary() { + log "" + log "Done. What to put in the control plane's environment:" + log "" + log " AWS_REGION=$REGION" + log " KMS_PROVIDER=aws" + log " KMS_AWS_KEY_ID=alias/$PREFIX" + log " BLOB_PROVIDER=s3" + log " BLOB_BUCKET=${BUCKET:-}" + log "" + log "Still to do by hand, because each one is a decision rather than a default:" + log "" + log " 1. an access key for $PREFIX-control, into the control plane's environment" + log " 2. the RDS security group: allow 5432 from your container host and your" + log " own address, not from everywhere" + log " 3. rds.force_ssl=1 in the instance's parameter group" + log " 4. the DKIM records above, in DNS" + log " 5. SES production access; a sandboxed account only delivers to verified" + log " addresses" + if [ -n "${DB_PASSWORD_PRINTED:-}" ]; then + log "" + log " The database master password, shown once:" + log " $DB_PASSWORD_PRINTED" + fi + log "" +} + +main() { + parse_args "$@" + require_args + create_kms + create_bucket + create_users + create_db + create_ses_identity + summary + return 0 +} + +main "$@" diff --git a/scripts/check-join-script.sh b/scripts/check-join-script.sh index 41849078..b5ce93b3 100755 --- a/scripts/check-join-script.sh +++ b/scripts/check-join-script.sh @@ -16,6 +16,8 @@ # systemd variable, not a command substitution systemd would never expand # - the mount list always includes the agent directory # - a relative BLOB_FS_ROOT is refused rather than mounted +# - the local override env file is passed AFTER node.env, so it wins, and is +# created without ever truncating one that is already there set -eu SCRIPT="internal/api/handler/nodescript/join.sh" @@ -85,6 +87,15 @@ printf '%s\n' "$unit" | grep -q 'ExecStart=.*-v /var/lib/warmbly/node:/var/lib/w || fail "the agent directory must always be mounted, or auto-update stops silently" ok "rendered unit (no blob mount)" +# node.local.env is the operator's half of the config and the only one a +# re-join does not rewrite. Docker applies --env-file in order, so it has to +# come AFTER node.env or an override silently loses to the generated value. +# Matched as one ordered pattern rather than two greps, which would pass with +# the files reversed. +printf '%s\n' "$unit" | grep -q 'ExecStart=.*--env-file /etc/warmbly/node.env .*--env-file /etc/warmbly/node.local.env' \ + || fail "node.local.env must be passed after node.env, or a local override loses to the generated value" +ok "local override env file is passed last" + # With local blobs the root has to be mounted too, and the line must still be # one line: a multi-line mount list is how the continuation collapsed before. unit=$(NODE_ENV="$FS_BLOB_ENV" sh "$SCRIPT" --print-unit) || fail "--print-unit with blobs failed" @@ -156,4 +167,26 @@ printf '%s\n' "$install_body" | awk '$1 == "ensure_blob_root" { found = 1 } END || fail "install_units must call ensure_blob_root as a standalone statement, or a filesystem-blob node restart-loops" ok "blob root is prepared before the unit is installed" +# The unit names node.local.env, so a join that does not create it leaves the +# service unable to start at all: docker refuses a missing --env-file. +write_body=$(body_of write_config) +printf '%s\n' "$write_body" | awk '$1 == "ensure_local_env" { found = 1 } END { exit !found }' \ + || fail "write_config must call ensure_local_env as a standalone statement; the unit names the file and docker refuses a missing --env-file" +ok "local override env file is created on join" + +# The whole point of the file is that a re-join keeps it. A creation path that +# can truncate would discard the credential an operator put there, which is +# both silent and unrecoverable. +# Matched on the shape of the invariant rather than one spelling of it: some +# existence test naming the file, before the line that writes it. `if [ -f ]`, +# `if ! test -f` and an AND-OR all satisfy this; only dropping the guard does +# not. +local_body=$(body_of ensure_local_env) +printf '%s\n' "$local_body" | awk ' + /-f .*node\.local\.env/ { guard = NR } + /> *"?\$CONFIG_DIR\/node\.local\.env"?/ { write = NR } + END { exit !(guard && write && guard < write) }' \ + || fail "ensure_local_env must test for an existing file before writing one, or a re-join truncates the operator's file" +ok "an existing local override file is never truncated" + printf 'check-join-script: all checks passed\n'