orca/cloud/docs/push-database-cutover.md

# Dedicated push database cutover

The gateway has a dedicated PostgreSQL 17 instance configured in
`infra/terraform/push-dedicated-database.tf`: regional HA, 2 vCPU, 7.5 GiB RAM, 50 GiB SSD
with automatic growth, seven retained backups and seven-day point-in-time recovery.
Cloud SQL and Terraform both protect it from deletion. Connections use the Cloud SQL
connector and a separate Secret Manager secret pinned to its Terraform-managed version.

Provisioning (`push_dedicated_database_enabled`) and attachment
(`push_dedicated_database_active`) are separate switches. Neither changes the existing
shared database or its credentials. The production feature is still in internal testing;
its owner approved an empty database and discarding the previous test state. No data
transfer or application maintenance mechanism is needed for this initial activation.
Test phones must register again; pending notifications and old sessions do not transfer.
This reset procedure is not suitable after public launch without explicit data-loss approval.

## Provision

Use the production relay backend and environment variables documented in
`infra/terraform/README.md`. Save and review a targeted Terraform plan containing only:

- `google_sql_database_instance.push_dedicated`
- `google_sql_database.push_dedicated`
- `random_password.push_dedicated_database`
- `google_sql_user.push_dedicated`
- `google_secret_manager_secret.push_dedicated_database_url`
- `google_secret_manager_secret_version.push_dedicated_database_url`
- `google_secret_manager_secret_iam_member.push_dedicated_database_url_accessor`

Require exactly seven additions and no updates or deletions for initial provisioning.
Apply that saved plan with backend locking, then require the same targeted plan to be
empty. Keep sensitive Terraform plans access-restricted; never print secret values or
upload raw state/plan JSON. Verify the instance is RUNNABLE with the expected version,
tier, backup policy, and regional availability. No Cloud Run service changes in this step.

## Activate the empty store

1. Record the current immutable serving image, revision, SQL attachment, and database
   secret reference. Confirm traffic is pinned to that revision rather than LATEST.
2. Set `push_dedicated_database_active = true` in production. Save a targeted plan for
   `google_cloud_run_v2_service.push`. Inspect its dependency closure and reject unrelated
   changes. Require only the SQL attachment and database secret reference to change.
3. Hold the existing Cloud SQL rollout lease while applying that saved service-shape plan.
   Terraform ignores traffic and image; verify traffic remains pinned to the old revision.
   Do not apply a plan that would shift traffic or revert runtime configuration.
4. Dispatch `cloud-push-deploy.yml` from main with the exact reviewed source SHA. It creates
   an inert, read-only candidate inheriting the dedicated attachment, probes readiness and provider
   access, then deliberately creates an active successor of the same digest before deleting validation.
   Activation starts schema writes and workers before HTTP promotion. Verify the candidate's SQL
   attachment and pinned secret reference as well as its image and health.
5. Register a test phone against the deployed origin and prove real APNs delivery. Check
   database errors and confirm the old revisions have no traffic or tags and source SQL
   connections have drained. Leave the old database intact; do not delete shared resources.

If activation fails before promotion, the existing HTTP serving revision is unchanged, but
activated workers may already have sent notifications or mutated the queue. Cloud Run will not
delete the latest created revision, even untagged at zero traffic. Recovery creates a known-good
successor first, verifies its template/runtime/secret shape and health, promotes and verifies it,
then deletes rejected and previous consumers. The recovery successor remains serving; it can run
known-good schema/workers before promotion and does not undo earlier queue or schema changes.
A partial activation leaving three resources must retire non-latest inert validation before
recovery creates another; failed retirement stops automation. Every deploy requires a single
serving revision resource at admission, so review and retire historical/leftover revisions under
the lease before dispatch. A Terraform attachment update can itself create such a revision:
verify/promote that known-good image and attachment and retire the former revision before dispatch.
The deploy workflow can roll traffic back on failure; in this internal reset rollout,
that may discard registrations created during the probe window. After successful activation,
application rollback should retain the dedicated attachment and deploy an older compatible
image through the workflow. Returning to the shared store is another explicit state reset,
not a lossless rollback. Future public migrations require a separately rehearsed transfer.

## Capacity and resizing

The initial gateway keeps its existing two-connection pool and two-instance maximum.
Dedicated database rollout pools are capped at 64 total configured pool connections across three simultaneous
revision resources (serving, validation/rejected, and active/recovery successor), leaving room for maintenance and operators; this is an admission
budget, not a throughput claim. Increase the pool only after measuring deployed contention.
Keep the shared database allocation reserved until source connections have drained.

Cloud SQL CPU/RAM resizing is an in-place infrastructure change but can interrupt database
connections. HA does not make a resize interruption-free. Durable accepted events remain in
SQL and workers retry after recovery within their five-minute expiry; requests that never
reach durable acceptance depend on client retries. Schedule resizes and verify reconnection,
queue recovery, readiness, and real delivery afterward.