mirror of
https://github.com/stablyai/orca.git
synced 2026-09-21 16:02:20 +00:00
Jinwoo Hong
5b8ac36f41
* feat(relay): report a post-wave health verdict on each same-cap cell, without gating on it After a same-cap cell finishes rolling, an operator reads five things by hand before dispatching the next cell: director 503s against the same clock hour a day and two days earlier, whether the cell's new container announced its listener and has stayed up, the cell's own pool pressure, the asia-east2 pool trio, and Cloud SQL FATALs. This runs those same reads automatically and records PASS / WARN / WOULD_BLOCK with its numbers, so its calls can be compared with the operator's over a full roll before it is ever allowed to stop one. It cannot fail a cell in this change. The script exits 0 on every verdict, and the step is continue-on-error, so even a crash stays off the job's outcome and the failure failsafe cannot fire on anything it observes. It also runs after the restore, so no cell waits on it to go back into admission. Cloud Logging returns only --limit entries and says nothing when it truncates, so every count is split into sub-windows of ten minutes and a sub-window that comes back at the limit is reported unverified rather than as a count. Windows are always explicitly bounded: --freshness does not bind on these logs. Claude-Session: https://claude.ai/session/ced32ebb-7155-4413-adad-1eccd14c2010 * fix(relay): bound the shadow gate's cell reads at the apply start and cap every read Four fixes from review, all in the report-only shadow health gate. The boot search opened at apply-completed-at, which is stamped after `terraform apply` and `wait-until --stable`. The new container announces its listener while the MIG is still converging, so that bound is already past the announcement it looks for and a healthy roll read as would-block. The job now stamps apply-started-at immediately before the apply, and the boot search opens there; apply-completed-at is kept, recorded rather than judged, so an operator comparing verdicts can see apply time next to boot time. The crash query started at the newest listener timestamp, which erased any crash before it. A crash-restart loop ends with an announcement that looks like a clean boot, so that is exactly the case it hid: against production, the 2026-09-20 c28 crash at 20:18:10 was dropped because the listener landed at 20:18:27. It now runs from the apply start, still scoped to the instance id the listener identified, and that crash is counted. A runtime-metrics read that came back at its 500-entry limit fed judgePool as though it were a complete sample run. A truncated run has holes and the consecutive-sample rule reads a hole as a recovery, so it now reports unverified. gcloud reads had no timeout. continue-on-error bounds the job's outcome but not its clock, so a stalled read could have spent the rollout's remaining minutes. Each read now gets 60 s and a timed-out read is just a failed read. Claude-Session: https://claude.ai/session/ced32ebb-7155-4413-adad-1eccd14c2010 * test(relay): require each shadow-gate stamp's presence before asserting its order The ordering assertion used indexOf, which answers -1 for an absent stamp, and -1 precedes every real offset. Deleting the apply-started-at line left the test green, so the census could not see the fix it was written to pin. Each stamp's presence is now asserted first, with a message naming the stamp and the step, and presence is judged inside the step that owns the stamp rather than anywhere in the file: a stamp written into a neighbouring step records the wrong instant but would satisfy a whole-file match. Control-run against a scratch copy of the job. Deleting drain-started-at, apply-started-at, or apply-completed-at each reds with its own message, and moving apply-started-at after terraform apply reds on the ordering assertion, so presence and order both fail independently. Claude-Session: https://claude.ai/session/ced32ebb-7155-4413-adad-1eccd14c2010
34 lines
4.7 KiB
JSON
34 lines
4.7 KiB
JSON
{
|
|
"name": "orca-cloud",
|
|
"private": true,
|
|
"version": "0.0.0",
|
|
"packageManager": "pnpm@10.24.0",
|
|
"engines": {
|
|
"node": ">=24 <27",
|
|
"pnpm": ">=10"
|
|
},
|
|
"scripts": {
|
|
"build": "pnpm -r build",
|
|
"dev": "pnpm --filter @orca-cloud/relay dev",
|
|
"incident:relay": "pnpm --filter @orca-cloud/relay-ops incident:monitor",
|
|
"incident:relay-preflight": "pnpm --filter @orca-cloud/relay-ops incident:preflight",
|
|
"infra:apply": "node dev/scripts/infra.mjs apply",
|
|
"infra:init": "node dev/scripts/infra.mjs init",
|
|
"infra:plan": "node dev/scripts/infra.mjs plan",
|
|
"lint": "pnpm -r lint",
|
|
"load:relay:controls": "node dev/scripts/load-relay-controls.mjs",
|
|
"load:relay:model": "node dev/scripts/run-relay-load-model.mjs",
|
|
"load:relay:recovery-gate": "node dev/scripts/run-relay-recovery-wave-gate.mjs",
|
|
"ops:relay": "pnpm --filter @orca-cloud/relay-ops dev",
|
|
"pretest": "node --test dev/scripts/capture-terraform-plan-baseline.test.mjs dev/scripts/operate-relay-asia-admission.test.mjs dev/scripts/prepare-relay-asia-director-cells.test.mjs dev/scripts/prepare-relay-asia-topology-input.test.mjs dev/scripts/production-cloud-sql-rollout-lock.test.mjs dev/scripts/read-relay-serving-regional-placement-version.test.mjs dev/scripts/relay-asia-admission-workflow.test.mjs dev/scripts/relay-asia-rollout-evidence.test.mjs dev/scripts/relay-asia-topology-workflow.test.mjs dev/scripts/relay-cloud-sql-connection-budget.test.mjs dev/scripts/relay-load-reader-evidence.test.mjs dev/scripts/relay-region-hint-metrics.test.mjs dev/scripts/relay-staging-deploy-identity.test.mjs dev/scripts/sanitize-relay-asia-admission-result.test.mjs dev/scripts/terraform-root-partition.test.mjs dev/scripts/validate-relay-asia-topology-plan.test.mjs ../.github/actions/cloud-sql-rollout-lease/action-contract.test.mjs ../.github/actions/cloud-sql-rollout-lease/storage-lease.test.mjs",
|
|
"test": "pnpm -r test && node --test dev/scripts/classify-relay-production-capacity-director.test.mjs dev/scripts/classify-relay-staging-bootstrap.test.mjs dev/scripts/deploy-relay-blue-green.test.mjs dev/scripts/deploy-relay-gce-candidate.test.mjs dev/scripts/deploy-relay-gce-multi-target.test.mjs dev/scripts/github-smoke-token.test.mjs dev/scripts/infra.test.mjs dev/scripts/operate-relay-regional-rehome.test.mjs dev/scripts/power-staging-relay.test.mjs dev/scripts/prepare-relay-capacity-canary.test.mjs dev/scripts/prepare-relay-production-capacity-canary.test.mjs dev/scripts/probe-relay-legacy-admission.test.mjs dev/scripts/probe-relay-rehome-trust.test.mjs dev/scripts/production-cell-image-digest-consistency.test.mjs dev/scripts/push-gateway-workflow.test.mjs dev/scripts/push-gateway-recovery.test.mjs dev/scripts/push-validation-workflow.test.mjs dev/scripts/read-relay-production-capacity-identity.test.mjs dev/scripts/relay-admin-endpoint-retry-workflow.test.mjs dev/scripts/relay-admin-transient-retry.test.mjs dev/scripts/relay-admission-selector.test.mjs dev/scripts/relay-gce-terraform-fence.test.mjs dev/scripts/relay-load-connection-failure.test.mjs dev/scripts/relay-load-control-peer.test.mjs dev/scripts/relay-load-director-capacity-gate.test.mjs dev/scripts/relay-load-model.test.mjs dev/scripts/relay-load-phase-barrier.test.mjs dev/scripts/relay-load-placement-boundary.test.mjs dev/scripts/relay-load-profile.test.mjs dev/scripts/relay-load-rebind-boundary.test.mjs dev/scripts/relay-load-region-behavior.test.mjs dev/scripts/relay-load-request-unit-boundary.test.mjs dev/scripts/relay-load-run-lifecycle.test.mjs dev/scripts/relay-monitor-evidence.test.mjs dev/scripts/relay-production-capacity-wave.test.mjs dev/scripts/relay-production-capacity-workflow.test.mjs dev/scripts/relay-production-identity-boundaries.test.mjs dev/scripts/relay-production-same-cap-wave.test.mjs dev/scripts/relay-public-workflow-contract.test.mjs dev/scripts/relay-recovery-wave-gate.test.mjs dev/scripts/relay-region-observation-evidence.test.mjs dev/scripts/relay-regional-rehome-workflow.test.mjs dev/scripts/relay-rehome-aggregate-evidence.test.mjs dev/scripts/relay-repository.test.mjs dev/scripts/relay-same-cap-script-census.test.mjs dev/scripts/relay-same-cap-shadow-gate.test.mjs dev/scripts/relay-staging-c4-refresh-workflow.test.mjs dev/scripts/relay-staging-capacity-identity.test.mjs dev/scripts/staging-relay-apply-guard.test.mjs dev/scripts/validate-relay-capacity-plan.test.mjs dev/scripts/verify-relay-capacity-transition.test.mjs dev/scripts/verify-relay-legacy-bootstrap.test.mjs dev/scripts/workload-identity-attribute-conditions.test.mjs",
|
|
"typecheck": "pnpm -r typecheck"
|
|
},
|
|
"devDependencies": {
|
|
"@types/node": "^24.10.0",
|
|
"tsx": "^4.21.0",
|
|
"typescript": "^5.9.3",
|
|
"vitest": "^4.1.11"
|
|
}
|
|
}
|