mirror of
https://github.com/stablyai/orca.git
synced 2026-10-30 16:47:34 +00:00
Jinwoo Hong
2bdb55350d
* fix(relay): director reaches Cloud SQL through a proxy sidecar (staging), gated event-loop stall alert On Cloud Run's built-in /cloudsql socket a connect() blocks the director's whole event loop until the connector's upstream dial ends, 10 s per attempt while the instance starts. Staging froze both directors for 134 s on 2026-10-11 and production four directors for 40-50 s on 2026-09-28. relay_director_cloud_sql_proxy_sidecar runs the Cloud SQL Auth Proxy beside the director on 127.0.0.1:5432, as the GCE cells already do, with its own TCP database URL secret. relay_event_loop_stall_alert adds the event_loop_ms_max metric and a 5 s stall alert. Both default off, so production plans unchanged; staging turns both on. * fix(relay): adopt a reviewed Terraform director shape on deploy; counter-based stall alert; cheaper probed sidecar - deploy-relay-blue-green: --shape-baseline-revision names the 0% revision a Terraform apply created; the deploy preserves that shape instead of the serving one. It must be the latest created, unserved revision on the serving image. Wired into the staging and production director workflows behind ADOPT_RELAY_DIRECTOR_TERRAFORM_SHAPE. - Event-loop stall alert is now a gated counter metric on eventLoopDelayMsMax>=5000. - Sidecar gets a /liveness probe and 0.25 vCPU (Cloud Run rejects only a total under 1 vCPU with concurrency above 1; checked with validateOnly against the API). - staging.tfvars pins the serving director's concurrency 1000 and timeout 3600 s, so the adopted apply shape does not fall to the 80 / 30 s defaults. * fix(relay): shape adoption refuses identity, topology, pruning and image changes; GOMAXPROCS=1 on the sidecar parseArguments now refuses --shape-baseline-revision with a runtime identity bootstrap, a director topology change or revision pruning, so every caller is covered, and the deploy refuses a shape baseline whose image differs from the serving one. Both director workflows also refuse the combinations up front. The proxy sidecar runs with GOMAXPROCS=1 to match its 0.25 vCPU quota. The long-hold alert comment is back above its resource.