Files
warmbly/deploy/terraform/aws/runtime
..

Application runtime

This root reads the foundation state and creates application services in us-east-1. Keep installation-specific inputs outside the checkout. Supply digest-pinned ARM64 images, valid Fargate sizes, desired task counts, secret key names and immutable secret version IDs. Secret values never enter Terraform.

terraform -chdir=deploy/terraform/aws/runtime init \
  -backend-config="bucket=$STATE_BUCKET"
terraform -chdir=deploy/terraform/aws/runtime plan \
  -var-file=/private/path/runtime.tfvars.json \
  -out=/private/path/runtime.tfplan
terraform -chdir=deploy/terraform/aws/runtime apply /private/path/runtime.tfplan

Start at desired count zero. Separate execution roles retrieve each service's configuration and write its logs. Application roles receive reviewed existing permissions or IAM Policy Autopilot output through application_policies.

Deploy releases

Terraform owns each task definition's shape: environment, secret references, sizes, health checks and the warmbly-image-variant tag that says whether a service runs the default or the -kafka image. Service revision and desired count are deployment operations; Terraform ignores those two fields after service creation, and its task definition outputs describe prepared revisions, not necessarily running ones. Start new services at zero and use explicit counts when activating them.

.github/workflows/deploy-aws.yml owns only the image. A stable release tag on main reaches it from release.yml when the repository variable AWS_DEPLOY_ENABLED is true; dispatch it by hand to redeploy, roll back or roll out changed secret values. Each run verifies the release's build provenance, renders every service from its latest Terraform revision with the release's digests, runs the dependency validation and migration tasks, then rolls backend, consumer, and forms, tracking and realtime in waves. Each service is watched against its CloudWatch alarms for a bake after it is stable, and any failure restores every service already moved, in reverse. Workers and consumers on hosts follow through a canary rollout started from the ops task. The procedure, modes and one-time GitHub setup are in docs/content/docs/development/aws-fargate.mdx.

Set github_repository to the repository running the workflows. The root creates an OIDC provider and a deployment role that can register task definition revisions, run the validation, migration, ops and rotator tasks, update the five services and read secret metadata, never a secret value. Import an existing account-wide GitHub OIDC provider instead of creating a second one. Set the aws-production environment's variables:

Variable Value
AWS_DEPLOY_ROLE_ARN Terraform github_deploy_role_arn output
AWS_ECS_CLUSTER Runtime cluster_arn output
AWS_ECS_SERVICE_PREFIX Runtime name input

Prepare a backend secret version with PRIMARY_DB_DIRECT, pointing to port 5432. The migration task injects only that key, with no application task role. The workflow never retrieves database credentials. Its backend and migration tasks must use the same immutable image digest and direct secret reference. Keep the default migration_secret_key=PRIMARY_DB_DIRECT for workflow deployments.

The migration runs /app/migrate inside the VPC over the existing PrivateLink route, and an unsuccessful or unresolved migration blocks every service update. A rollback skips it, because schemas stay compatible with the previous release. The workflow never runs down migrations. Task definitions use skip_destroy so earlier revisions remain active for rollback. Retain their referenced images and secret versions throughout the rollback window; retire them separately after it closes.

Rotate keys

rotation.tf registers a rotator task that generates replacement values with its own narrow role, writes them as new secret versions and reports only version ids. .github/workflows/rotate-keys.yml drives it, rolls services onto those versions and runs warmblyctl in the ops task to prove each step before the old value is retired. No value passes through GitHub or a log. The runbook is in aws-fargate.mdx.

Terraform in CI

Set terraform_ci = true to create the <name>-terraform-plan and <name>-terraform-apply roles .github/workflows/terraform-aws.yml uses. The plan role reads configuration and is denied secret values, decryption, logs and data reads; the apply role administers resources named for this deployment and is reachable only from main through protected environments. The apply role cannot change the two CI roles themselves, so a change to their policies or trust is applied by an administrator by hand. Var files stay in the state bucket under warmbly/production/vars/; only resource addresses and actions are ever printed.

The runtime sets DB_MIGRATIONS_ON_START=false for the backend. New application images must support that setting and schema readiness before activation. Keep migrations additive and compatible with the previous release throughout rollout and rollback. Do not interrupt an active migration; if a workflow is cancelled or times out, inspect the ECS task before rerunning it.

Run database_check_task_definition to exercise the backend image's actual Go client against its configured pooler without starting the API or background jobs. It collects operator status into temporary task storage and logs only completion. The separate dependency validation task checks provider connectivity and stored credential decryption. Neither check activates the backend service.

Tasks use public IPs for outbound access, avoiding NAT gateway charges. Inbound application traffic is restricted to the HTTPS load balancer. The backend also accepts private task traffic through service discovery. Restrict the load balancer to approved caller CIDRs, such as Cloudflare's published ranges. Public DNS changes are separate from deploying the services.

Run validation_task_definition once in a public subnet with a public IP and the task security group. It checks private database DNS and logins, stored mailbox and MFA decryption, KMS, both caches, Kafka metadata, Schema Registry and S3 using the actual backend task role. It writes no production data, logs only counts and results, and exits. Observe its stopped exit code and logs before activation. The pinned validation packages are installed in temporary task storage.

Keep one cache authoritative throughout deployment overlap. A backend starts background jobs immediately; its desired count is the activation control. Preserve compatible images, encryption material and shared coordination while the previous deployment drains. Verify target health and HTTPS routes before switching DNS. New injected values require replacing tasks.