starred/orca
1
0
Fork 0
mirror of https://github.com/stablyai/orca.git synced 2026-10-08 16:02:37 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
4e64fa9940f5a4a9bfb7b021a3c405fd76643ca5
orca/.github/workflows/cloud-deploy-relay-production-same-cap.yml
T
Jinwoo Hong f1ed355d06 feat(relay): drain pace window as a reviewed same-cap input, with drain-aware 503 gates (#25639) 
* feat(relay): drain pace window as a reviewed same-cap input, with drain-aware 503 gates

The same-cap roll drained every cell over a fixed 300 s window, so a US roll
re-placed hosts at ~2/s and spent ~10 minutes draining and waiting for quiet.
The window is now a dispatch input from a closed set (300000, 60000, 30000),
defaulting to today's 300000.

- Below the default is refused for anything but US general cells; Asia drains
  are bound by their targets' accept rate, and migration-only cells hold no hosts.
- A non-default window must be named in the confirmation, the canary authority
  records it (v2), and a batch may run at its canary's window or slower only.
- Each cell job re-checks the window, scales the restart-safe timeout with it
  (15-min lease + window, unchanged at the default), and records what the cell
  applied and when it settled.
- The report-only shadow gate takes the director's drain-return deferrals out
  of the 503 count (window and baselines), adds the rung's 5-min non-drain 503
  budget and a Retry-After check, and reports the measured re-placement rate.
- relay-workflows.md documents the pace ladder and what each rung records.

* fix(relay): judge paced drains on counted 503s against the pre-drain minutes, and seal the canary's pace verdict

Review of #25639 replayed the shadow gate: it read 10-01 c29 as unverified (the
log read stopped at 20k entries), false-blocked 10-02 c22, and was blind on four
cells whose 24 h/48 h baseline held an incident.

- Director 503s now come from Cloud Run's request_count, aligned per minute by
  Cloud Monitoring, so volume cannot truncate the count.
- The background is the median of the 10 same-day minutes before the drain;
  the 24 h/48 h baselines are gone.
- Scheduled 503s come out: drain-return deferrals and sticky/placement answers
  to a host's own early retry (host-rate-limited, host-in-flight), each split
  across the minutes its 30 s sample covers.
- The rung budget counts only sustained breaches: two straight minutes over
  max(1.5x, +20) warn, over max(2x, +40) would-block.
- The report carries a paceVerdict over the three pace checks. seal_canary
  downloads the canary cell's report and seals that verdict; a batch below the
  default pace needs PASS, from a report on the same cell that drained at that
  pace.
- Docs: the step-down rule reads paceVerdict, 30 s waits for the lane service
  time (#25645), and the staging step is dropped since staging drains unpaced.

Replayed read-only: 10-01 c29 would-block (9 minutes over 41.5/min); all nine
10-02 cells and 10-01 c25 paceVerdict PASS.

* fix(relay): a partial count already past a block line blocks, in the shadow gate's Cloud SQL and pool checks

A truncated FATAL count is a floor, and one runtime sample over the SQL-failure
line is a fact, so neither waits for a complete read. The waiter-run rule still
needs a complete run, since holes can join two runs into one.

* fix(relay): a canary pace PASS needs a real cohort and whole telemetry; one median-based 503 check

From the final review of #25639:
- canaryPaceVerdict seals PASS only from a report that drained at least 400
  hosts (about half a 10-02 US cell), so a near-empty canary cannot authorize
  a fast batch.
- A director-metrics sub-window with fewer samples than one instance emits
  is unverified, so an empty or short Logging answer is never a calm drain.
- seal_canary names the shadow artifact, report path and cell from the gate's
  normalized cell list, as cell_1 uploads it.
- director503 folds into nonDrain503Budget as a single-minute spike rule,
  max(10x median, 200), dropping the pre-drain peak statistic. All 30 replayed
  windows keep their verdicts.
2026-10-05 17:31:29 -04:00

432 lines
19 KiB
YAML
Raw Blame History

name: Deploy Relay Production Same-Cap
on:
workflow_dispatch:
inputs:
mode:
description: Verify, roll one canary, roll a bounded batch, or roll back
required: true
default: verify
type: choice
options: [verify, canary-apply, batch-apply, rollback]
cell-ids:
description: Ordered comma-separated serving cells; one canary or two to ten batch cells
required: true
type: string
target-image-digest:
description: Exact immutable compatibility image digest
required: true
type: string
rollback-image-digest:
description: Exact immutable currently serving rollback digest
required: true
type: string
target-rehome-protocol:
description: Exact target regional-rehome protocol
required: true
default: '1'
type: choice
options: ['0', '1', '3']
rollback-rehome-protocol:
description: Exact rollback regional-rehome protocol
required: true
default: '0'
type: choice
options: ['0', '1', '3']
expected-selector-generation:
description: Exact selector generation before the first cell
required: true
type: string
expected-existing-only-cells:
description: Exact existing-only membership, or none
required: true
type: string
expected-migration-only-cells:
description: Exact migration-only membership, or none
required: true
type: string
expected-general-cells:
description: Exact general membership, or none
required: true
type: string
expected-rehome-generation:
description: Exact durable regional-rehome control generation; it must be disabled
required: true
type: string
drain-pace-window-ms:
description: Window the cell spreads its drain over; below 300000 only for US general cells, named in the confirmation
required: true
default: '300000'
type: choice
options: ['300000', '60000', '30000']
canary-run-id:
description: Successful same-code canary in this rehome control generation; reusable across batches
required: false
type: string
confirmation:
description: Exact digest-and-cell-bound mutation confirmation
required: false
type: string
permissions:
actions: read
contents: read
id-token: write
concurrency:
group: production-cloud-sql-rollout
cancel-in-progress: false
defaults:
run:
working-directory: cloud
jobs:
gate:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' && (github.ref == 'refs/heads/main') }}
runs-on: blacksmith-2vcpu-ubuntu-2204
# Headroom for the full-history checkout the canary provenance check needs.
timeout-minutes: 15
environment: production
outputs:
cells: ${{ steps.wave.outputs.cells }}
job-mode: ${{ steps.wave.outputs.job-mode }}
steps:
# Full history: the canary authority a batch verifies is sealed at an ancestor commit, and
# the provenance check fails closed on a commit a shallow clone left out.
- uses: actions/checkout@v4
with:
fetch-depth: 0
- uses: actions/setup-node@v4
with: { node-version: 24 }
- id: wave
env:
MODE: ${{ inputs.mode }}
CELL_IDS: ${{ inputs.cell-ids }}
TARGET_DIGEST: ${{ inputs.target-image-digest }}
ROLLBACK_DIGEST: ${{ inputs.rollback-image-digest }}
CONFIRMATION: ${{ inputs.confirmation }}
CANARY_RUN_ID: ${{ inputs.canary-run-id }}
DRAIN_PACE_WINDOW_MS: ${{ inputs.drain-pace-window-ms }}
run: |
CELLS="$(node dev/scripts/relay-production-same-cap-wave.mjs validate \
--mode "${MODE}" --cell-ids "${CELL_IDS}" \
--target-digest "${TARGET_DIGEST}" --rollback-digest "${ROLLBACK_DIGEST}" \
--confirmation "${CONFIRMATION}" --canary-run-id "${CANARY_RUN_ID}" \
--drain-pace-window-ms "${DRAIN_PACE_WINDOW_MS}")"
echo "cells=${CELLS}" >> "${GITHUB_OUTPUT}"
if [[ "${MODE}" =~ ^(canary-apply|batch-apply)$ ]]; then
echo 'job-mode=apply' >> "${GITHUB_OUTPUT}"
else
echo "job-mode=${MODE}" >> "${GITHUB_OUTPUT}"
fi
- name: Download exact prior canary authority
if: ${{ inputs.mode == 'batch-apply' }}
uses: actions/download-artifact@v4
with:
name: relay-same-cap-canary-${{ inputs.canary-run-id }}
path: ${{ runner.temp }}/relay-same-cap-canary
github-token: ${{ github.token }}
run-id: ${{ inputs.canary-run-id }}
- name: Verify canary authority against this batch
if: ${{ inputs.mode == 'batch-apply' }}
env:
CANARY_RUN_ID: ${{ inputs.canary-run-id }}
CELL_IDS: ${{ inputs.cell-ids }}
TARGET_DIGEST: ${{ inputs.target-image-digest }}
ROLLBACK_DIGEST: ${{ inputs.rollback-image-digest }}
SELECTOR_GENERATION: ${{ inputs.expected-selector-generation }}
REHOME_GENERATION: ${{ inputs.expected-rehome-generation }}
DRAIN_PACE_WINDOW_MS: ${{ inputs.drain-pace-window-ms }}
run: |
node dev/scripts/relay-production-same-cap-wave.mjs verify-canary \
--file "${RUNNER_TEMP}/relay-same-cap-canary/authority.json" \
--commit-sha "${GITHUB_SHA}" --run-id "${CANARY_RUN_ID}" \
--cell-ids "${CELL_IDS}" \
--target-digest "${TARGET_DIGEST}" \
--rollback-digest "${ROLLBACK_DIGEST}" \
--drain-pace-window-ms "${DRAIN_PACE_WINDOW_MS}" \
--selector-generation "${SELECTOR_GENERATION}" \
--rehome-generation "${REHOME_GENERATION}"
cell_1:
needs: gate
uses: ./.github/workflows/cloud-deploy-relay-production-same-cap-job.yml
with:
mode: ${{ needs.gate.outputs.job-mode }}
target-cell-id: ${{ fromJSON(needs.gate.outputs.cells)[0] }}
target-image-digest: ${{ inputs.target-image-digest }}
rollback-image-digest: ${{ inputs.rollback-image-digest }}
target-rehome-protocol: ${{ inputs.target-rehome-protocol }}
rollback-rehome-protocol: ${{ inputs.rollback-rehome-protocol }}
expected-selector-generation: ${{ inputs.expected-selector-generation }}
expected-existing-only-cells: ${{ inputs.expected-existing-only-cells }}
expected-migration-only-cells: ${{ inputs.expected-migration-only-cells }}
expected-general-cells: ${{ inputs.expected-general-cells }}
expected-rehome-generation: ${{ inputs.expected-rehome-generation }}
drain-pace-window-ms: ${{ inputs.drain-pace-window-ms }}
wave-index: '0'
secrets: inherit
cell_2:
if: ${{ needs.cell_1.result == 'success' && fromJSON(needs.gate.outputs.cells)[1] != null }}
needs: [gate, cell_1]
uses: ./.github/workflows/cloud-deploy-relay-production-same-cap-job.yml
with:
mode: ${{ needs.gate.outputs.job-mode }}
target-cell-id: ${{ fromJSON(needs.gate.outputs.cells)[1] }}
target-image-digest: ${{ inputs.target-image-digest }}
rollback-image-digest: ${{ inputs.rollback-image-digest }}
target-rehome-protocol: ${{ inputs.target-rehome-protocol }}
rollback-rehome-protocol: ${{ inputs.rollback-rehome-protocol }}
expected-selector-generation: ${{ inputs.expected-selector-generation }}
expected-existing-only-cells: ${{ inputs.expected-existing-only-cells }}
expected-migration-only-cells: ${{ inputs.expected-migration-only-cells }}
expected-general-cells: ${{ inputs.expected-general-cells }}
expected-rehome-generation: ${{ inputs.expected-rehome-generation }}
drain-pace-window-ms: ${{ inputs.drain-pace-window-ms }}
wave-index: '1'
secrets: inherit
cell_3:
if: ${{ needs.cell_2.result == 'success' && fromJSON(needs.gate.outputs.cells)[2] != null }}
needs: [gate, cell_2]
uses: ./.github/workflows/cloud-deploy-relay-production-same-cap-job.yml
with:
mode: ${{ needs.gate.outputs.job-mode }}
target-cell-id: ${{ fromJSON(needs.gate.outputs.cells)[2] }}
target-image-digest: ${{ inputs.target-image-digest }}
rollback-image-digest: ${{ inputs.rollback-image-digest }}
target-rehome-protocol: ${{ inputs.target-rehome-protocol }}
rollback-rehome-protocol: ${{ inputs.rollback-rehome-protocol }}
expected-selector-generation: ${{ inputs.expected-selector-generation }}
expected-existing-only-cells: ${{ inputs.expected-existing-only-cells }}
expected-migration-only-cells: ${{ inputs.expected-migration-only-cells }}
expected-general-cells: ${{ inputs.expected-general-cells }}
expected-rehome-generation: ${{ inputs.expected-rehome-generation }}
drain-pace-window-ms: ${{ inputs.drain-pace-window-ms }}
wave-index: '2'
secrets: inherit
cell_4:
if: ${{ needs.cell_3.result == 'success' && fromJSON(needs.gate.outputs.cells)[3] != null }}
needs: [gate, cell_3]
uses: ./.github/workflows/cloud-deploy-relay-production-same-cap-job.yml
with:
mode: ${{ needs.gate.outputs.job-mode }}
target-cell-id: ${{ fromJSON(needs.gate.outputs.cells)[3] }}
target-image-digest: ${{ inputs.target-image-digest }}
rollback-image-digest: ${{ inputs.rollback-image-digest }}
target-rehome-protocol: ${{ inputs.target-rehome-protocol }}
rollback-rehome-protocol: ${{ inputs.rollback-rehome-protocol }}
expected-selector-generation: ${{ inputs.expected-selector-generation }}
expected-existing-only-cells: ${{ inputs.expected-existing-only-cells }}
expected-migration-only-cells: ${{ inputs.expected-migration-only-cells }}
expected-general-cells: ${{ inputs.expected-general-cells }}
expected-rehome-generation: ${{ inputs.expected-rehome-generation }}
drain-pace-window-ms: ${{ inputs.drain-pace-window-ms }}
wave-index: '3'
secrets: inherit
cell_5:
if: ${{ needs.cell_4.result == 'success' && fromJSON(needs.gate.outputs.cells)[4] != null }}
needs: [gate, cell_4]
uses: ./.github/workflows/cloud-deploy-relay-production-same-cap-job.yml
with:
mode: ${{ needs.gate.outputs.job-mode }}
target-cell-id: ${{ fromJSON(needs.gate.outputs.cells)[4] }}
target-image-digest: ${{ inputs.target-image-digest }}
rollback-image-digest: ${{ inputs.rollback-image-digest }}
target-rehome-protocol: ${{ inputs.target-rehome-protocol }}
rollback-rehome-protocol: ${{ inputs.rollback-rehome-protocol }}
expected-selector-generation: ${{ inputs.expected-selector-generation }}
expected-existing-only-cells: ${{ inputs.expected-existing-only-cells }}
expected-migration-only-cells: ${{ inputs.expected-migration-only-cells }}
expected-general-cells: ${{ inputs.expected-general-cells }}
expected-rehome-generation: ${{ inputs.expected-rehome-generation }}
drain-pace-window-ms: ${{ inputs.drain-pace-window-ms }}
wave-index: '4'
secrets: inherit
cell_6:
if: ${{ needs.cell_5.result == 'success' && fromJSON(needs.gate.outputs.cells)[5] != null }}
needs: [gate, cell_5]
uses: ./.github/workflows/cloud-deploy-relay-production-same-cap-job.yml
with:
mode: ${{ needs.gate.outputs.job-mode }}
target-cell-id: ${{ fromJSON(needs.gate.outputs.cells)[5] }}
target-image-digest: ${{ inputs.target-image-digest }}
rollback-image-digest: ${{ inputs.rollback-image-digest }}
target-rehome-protocol: ${{ inputs.target-rehome-protocol }}
rollback-rehome-protocol: ${{ inputs.rollback-rehome-protocol }}
expected-selector-generation: ${{ inputs.expected-selector-generation }}
expected-existing-only-cells: ${{ inputs.expected-existing-only-cells }}
expected-migration-only-cells: ${{ inputs.expected-migration-only-cells }}
expected-general-cells: ${{ inputs.expected-general-cells }}
expected-rehome-generation: ${{ inputs.expected-rehome-generation }}
drain-pace-window-ms: ${{ inputs.drain-pace-window-ms }}
wave-index: '5'
secrets: inherit
cell_7:
if: ${{ needs.cell_6.result == 'success' && fromJSON(needs.gate.outputs.cells)[6] != null }}
needs: [gate, cell_6]
uses: ./.github/workflows/cloud-deploy-relay-production-same-cap-job.yml
with:
mode: ${{ needs.gate.outputs.job-mode }}
target-cell-id: ${{ fromJSON(needs.gate.outputs.cells)[6] }}
target-image-digest: ${{ inputs.target-image-digest }}
rollback-image-digest: ${{ inputs.rollback-image-digest }}
target-rehome-protocol: ${{ inputs.target-rehome-protocol }}
rollback-rehome-protocol: ${{ inputs.rollback-rehome-protocol }}
expected-selector-generation: ${{ inputs.expected-selector-generation }}
expected-existing-only-cells: ${{ inputs.expected-existing-only-cells }}
expected-migration-only-cells: ${{ inputs.expected-migration-only-cells }}
expected-general-cells: ${{ inputs.expected-general-cells }}
expected-rehome-generation: ${{ inputs.expected-rehome-generation }}
drain-pace-window-ms: ${{ inputs.drain-pace-window-ms }}
wave-index: '6'
secrets: inherit
cell_8:
if: ${{ needs.cell_7.result == 'success' && fromJSON(needs.gate.outputs.cells)[7] != null }}
needs: [gate, cell_7]
uses: ./.github/workflows/cloud-deploy-relay-production-same-cap-job.yml
with:
mode: ${{ needs.gate.outputs.job-mode }}
target-cell-id: ${{ fromJSON(needs.gate.outputs.cells)[7] }}
target-image-digest: ${{ inputs.target-image-digest }}
rollback-image-digest: ${{ inputs.rollback-image-digest }}
target-rehome-protocol: ${{ inputs.target-rehome-protocol }}
rollback-rehome-protocol: ${{ inputs.rollback-rehome-protocol }}
expected-selector-generation: ${{ inputs.expected-selector-generation }}
expected-existing-only-cells: ${{ inputs.expected-existing-only-cells }}
expected-migration-only-cells: ${{ inputs.expected-migration-only-cells }}
expected-general-cells: ${{ inputs.expected-general-cells }}
expected-rehome-generation: ${{ inputs.expected-rehome-generation }}
drain-pace-window-ms: ${{ inputs.drain-pace-window-ms }}
wave-index: '7'
secrets: inherit
cell_9:
if: ${{ needs.cell_8.result == 'success' && fromJSON(needs.gate.outputs.cells)[8] != null }}
needs: [gate, cell_8]
uses: ./.github/workflows/cloud-deploy-relay-production-same-cap-job.yml
with:
mode: ${{ needs.gate.outputs.job-mode }}
target-cell-id: ${{ fromJSON(needs.gate.outputs.cells)[8] }}
target-image-digest: ${{ inputs.target-image-digest }}
rollback-image-digest: ${{ inputs.rollback-image-digest }}
target-rehome-protocol: ${{ inputs.target-rehome-protocol }}
rollback-rehome-protocol: ${{ inputs.rollback-rehome-protocol }}
expected-selector-generation: ${{ inputs.expected-selector-generation }}
expected-existing-only-cells: ${{ inputs.expected-existing-only-cells }}
expected-migration-only-cells: ${{ inputs.expected-migration-only-cells }}
expected-general-cells: ${{ inputs.expected-general-cells }}
expected-rehome-generation: ${{ inputs.expected-rehome-generation }}
drain-pace-window-ms: ${{ inputs.drain-pace-window-ms }}
wave-index: '8'
secrets: inherit
cell_10:
if: ${{ needs.cell_9.result == 'success' && fromJSON(needs.gate.outputs.cells)[9] != null }}
needs: [gate, cell_9]
uses: ./.github/workflows/cloud-deploy-relay-production-same-cap-job.yml
with:
mode: ${{ needs.gate.outputs.job-mode }}
target-cell-id: ${{ fromJSON(needs.gate.outputs.cells)[9] }}
target-image-digest: ${{ inputs.target-image-digest }}
rollback-image-digest: ${{ inputs.rollback-image-digest }}
target-rehome-protocol: ${{ inputs.target-rehome-protocol }}
rollback-rehome-protocol: ${{ inputs.rollback-rehome-protocol }}
expected-selector-generation: ${{ inputs.expected-selector-generation }}
expected-existing-only-cells: ${{ inputs.expected-existing-only-cells }}
expected-migration-only-cells: ${{ inputs.expected-migration-only-cells }}
expected-general-cells: ${{ inputs.expected-general-cells }}
expected-rehome-generation: ${{ inputs.expected-rehome-generation }}
drain-pace-window-ms: ${{ inputs.drain-pace-window-ms }}
wave-index: '9'
secrets: inherit
seal_canary:
if: ${{ inputs.mode == 'canary-apply' }}
needs: [gate, cell_1]
runs-on: blacksmith-2vcpu-ubuntu-2204
timeout-minutes: 5
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with: { node-version: 24 }
# Its pace checks are what let a batch drain faster than the default; a missing artifact
# seals UNVERIFIED, which only the default accepts. Named from the gate's normalized cell,
# as cell_1 uploaded it, so a padded cell-ids input cannot miss it.
- name: Download the canary cell's shadow health verdict
continue-on-error: true
uses: actions/download-artifact@v4
with:
name: relay-same-cap-shadow-gate-${{ fromJSON(needs.gate.outputs.cells)[0] }}-${{ github.run_id }}.json
path: ${{ runner.temp }}/relay-same-cap-shadow-gate
- name: Seal exact successful canary authority
run: |
mkdir -p "${RUNNER_TEMP}/relay-same-cap-canary"
node dev/scripts/relay-production-same-cap-wave.mjs create-canary \
--cell-id "${{ fromJSON(needs.gate.outputs.cells)[0] }}" \
--target-digest "${{ inputs.target-image-digest }}" \
--rollback-digest "${{ inputs.rollback-image-digest }}" \
--confirmation "${{ inputs.confirmation }}" \
--drain-pace-window-ms "${{ inputs.drain-pace-window-ms }}" \
--shadow-report "${RUNNER_TEMP}/relay-same-cap-shadow-gate/relay-same-cap-shadow-gate-${{ fromJSON(needs.gate.outputs.cells)[0] }}-${GITHUB_RUN_ID}.json" \
--commit-sha "${GITHUB_SHA}" --run-id "${GITHUB_RUN_ID}" \
--selector-generation "${{ inputs.expected-selector-generation }}" \
--rehome-generation "${{ inputs.expected-rehome-generation }}" \
> "${RUNNER_TEMP}/relay-same-cap-canary/authority.json"
- uses: actions/upload-artifact@v4
with:
name: relay-same-cap-canary-${{ github.run_id }}
path: ${{ runner.temp }}/relay-same-cap-canary/authority.json
retention-days: 30
if-no-files-found: error
# Every cell job re-enters the run's lease with release: 'false'; only this job frees it.
release_lease:
if: always()
needs:
- gate
- cell_1
- cell_2
- cell_3
- cell_4
- cell_5
- cell_6
- cell_7
- cell_8
- cell_9
- cell_10
- seal_canary
runs-on: blacksmith-2vcpu-ubuntu-2204
timeout-minutes: 10
environment: production
steps:
- uses: actions/checkout@v4
- uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
- uses: google-github-actions/setup-gcloud@v2
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-terraform-state
object: terraform/state/cloud-sql-rollout/production.lock
release: 'true'
Reference in New Issue View Git Blame Copy Permalink