mirror of
https://github.com/stablyai/orca.git
synced 2026-11-02 10:08:13 +00:00
* feat(relay): watch a reserve cell and flip it back on the first step 5 stop rule A manual, inert-until-dispatched workflow samples one reserve cell every minute for up to 300 minutes against the step 5 §9 stop rules, logs one JSON line per sample, and on the first trip (or N unreadable samples in a row) runs the flag tool's own admitMode=db write under the flag workflow's concurrency group. dry-run logs the flag tool's plan instead. Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * fix(relay): keep the reserve watch's flip back out of the flag group, and baseline 503s before the flip - The flip back holds no concurrency group: GitHub keeps one pending run per group and a newer one cancels it. The flag tool's CAS orders it instead; a generation conflict is re-read and retried (5 tries, backoff) while the switch file still says reserve. - The 503 background comes from the 10 minutes before a required flipped-at, or an explicit value; neither is refused. - A failed or cancelled write-mode watch flips back too. An unreachable cell fails with the exact break-glass inputs. - Missing 503 data (empty series, unpublished minute, unparseable time) is unverifiable. - Seat-behind-row key adds the director instance and cell; six segments share one command. Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * fix(relay): read the reserve watch's 503 publishing front from every response code A quiet fleet has no 503 series, so the watch read it as unverifiable and flipped a healthy cell back within minutes. The 503 read now groups request_count by response code in one query; the newest minute any code has a point for is how far the series is published, so a 503 minute at or before it with no point is zero and minutes past it are left out. Also: the flip-back job's steps run on always(), flipped-at ages against the watch's start in later segments, a pending pre-flip background no longer adds to the unverifiable streak, the flip back retries runtime-status before calling for break glass, and the concurrency rationale lives once in the docs. Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * docs(relay): state what a cancel during the flip-back lease wait can leave, and the 503 publish-skew assumption Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * fix(relay): fail a segment closed on a corrupt watch state file, and keep no checkout token on disk Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * fix(relay): alarm on a failed flip back, refuse a watch under a rollout lease, wait for reserve to apply, and control the duplicate read - A flip back whose write or plan never finished (no ID token, a failed login, a silent cell) raises an ::error:: annotation and a FLIP BACK FAILED summary, as the clock-skew monitor does. - The first segment reads the Cloud SQL rollout lease through the lease action's own reader and refuses to start while any run holds it, or when it cannot be read. - A cell not yet reporting reserve is waited on for up to 10 minutes, judging nothing, before the watch ends refused without a write. - The duplicate-seat read takes the per-minute ledger summary in the same query and window; with no fresh summary in it the sample is unverifiable, not zero duplicates. Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * fix(relay): page on a refused reserve watch or a failed flip back A refused watch can leave a cell in reserve and unwatched, the same as a flip back that did not land. Both now raise an escaped ::error:: annotation through one `alarm` command, and in write mode write orca_relay_reserve_watch_alarm with gcloud logging write. A new log-based metric and alert policy page on any such line, per cell, through relay_alert_notification_channels, and the deploy identity gets roles/logging.logWriter in production to write it. Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * fix(relay): name the log-writer risk, check the alarm's project, and say a failed login cannot page Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * docs(relay): note a force-cancel can skip the reserve-watch flip-back write
181 lines
8.6 KiB
JavaScript
181 lines
8.6 KiB
JavaScript
import assert from 'node:assert/strict'
|
|
import { readFile } from 'node:fs/promises'
|
|
import test from 'node:test'
|
|
import { readRelayWorkflow, relayWorkflowFile } from './relay-repository.mjs'
|
|
import { readWorkflow, workflowFiles } from './cloud-sql-rollout-lock-census.mjs'
|
|
|
|
async function source(path) {
|
|
return await readFile(new URL(`../../${path}`, import.meta.url), 'utf8')
|
|
}
|
|
|
|
// Why: the shared deploy identity is relay-owned in production and moves with the relay
|
|
// extraction, so it needs a name the public repo can carry without touching the app pair. The
|
|
// generic names are retired; a workflow that still reads them would silently resolve to nothing.
|
|
test('no workflow names the retired generic production deploy identity', async () => {
|
|
const files = workflowFiles()
|
|
assert.ok(files.length > 20)
|
|
const relayReaders = []
|
|
for (const file of files) {
|
|
const workflow = readWorkflow(file)
|
|
assert.doesNotMatch(workflow, /PRODUCTION_GCP_WORKLOAD_IDENTITY_PROVIDER\b/, file)
|
|
assert.doesNotMatch(workflow, /PRODUCTION_GCP_DEPLOY_SERVICE_ACCOUNT\b/, file)
|
|
if (/PRODUCTION_GCP_RELAY_DEPLOY_/.test(workflow)) relayReaders.push(file)
|
|
}
|
|
assert.deepEqual(relayReaders.sort(), [
|
|
'deploy-relay-fence-broker.yml',
|
|
'deploy-relay-production-capacity-job.yml',
|
|
'deploy-relay-production-capacity.yml',
|
|
'deploy-relay-production-director.yml',
|
|
'deploy-relay-production-multi-target.yml',
|
|
'deploy-relay-production-same-cap-job.yml',
|
|
'deploy-relay-production-same-cap.yml',
|
|
'deploy-relay-production.yml',
|
|
'operate-relay-asia-admission.yml',
|
|
'operate-relay-production-cell-flags.yml',
|
|
'operate-relay-production-rehome-job.yml',
|
|
'operate-relay-production-reserve-watch.yml',
|
|
'publish-relay-production.yml'
|
|
].map((name) => relayWorkflowFile(name)).sort())
|
|
})
|
|
|
|
test('monitor workflow has no shared deploy identity fallback', async () => {
|
|
const workflow = readRelayWorkflow('monitor-relay-production-job.yml')
|
|
assert.match(workflow, /PRODUCTION_GCP_RELAY_MONITOR_WORKLOAD_IDENTITY_PROVIDER/)
|
|
assert.match(workflow, /PRODUCTION_GCP_RELAY_MONITOR_SERVICE_ACCOUNT/)
|
|
assert.doesNotMatch(workflow, /PRODUCTION_GCP_DEPLOY_SERVICE_ACCOUNT/)
|
|
assert.doesNotMatch(workflow, /PRODUCTION_GCP_WORKLOAD_IDENTITY_PROVIDER/)
|
|
})
|
|
|
|
test('relay fencing uses the dedicated requester and private broker', async () => {
|
|
const workflow = readRelayWorkflow('deploy-relay-production-multi-target.yml')
|
|
assert.match(workflow, /Reject direct-runner Terraform fence aborts/)
|
|
assert.match(workflow, /inputs\.mode == 'fence-source'/)
|
|
assert.match(workflow, /inputs\.mode == 'abort-fence-source'/)
|
|
assert.match(workflow, /inputs\.mode == 'supersede-target'/)
|
|
assert.match(workflow, /PRODUCTION_GCP_RELAY_FENCE_WORKLOAD_IDENTITY_PROVIDER/)
|
|
assert.match(workflow, /PRODUCTION_GCP_RELAY_FENCE_SERVICE_ACCOUNT/)
|
|
assert.match(workflow, /PRODUCTION_GCP_RELAY_FENCE_BROKER_URI/)
|
|
assert.match(workflow, /Invoke private target-supersession broker/)
|
|
assert.match(workflow, /Invoke private source-fence broker/)
|
|
assert.match(workflow, /Require exact broker cell contract/)
|
|
assert.match(workflow, /Require exact source-fence broker contract/)
|
|
assert.match(workflow, /Require private fence-broker environment/)
|
|
assert.match(
|
|
workflow,
|
|
/DEPLOY_MODE\}" = "execute" \|\|\s+"\$\{DEPLOY_MODE\}" = "recover-forward"\) &&\s+"\$\{SOURCE_CELL_ID\}" = "production-gce-c12"/
|
|
)
|
|
assert.match(
|
|
workflow,
|
|
/--scoped-recovery-source-cell-id\s+production-gce-c3/
|
|
)
|
|
assert.match(
|
|
workflow,
|
|
/test "\$\{FAILED_TARGET_CELL_ID\}" = "production-gce-c12"/
|
|
)
|
|
assert.match(
|
|
workflow,
|
|
/test "\$\{REPLACEMENT_TARGET_CELL_ID\}" = "production-gce-c13"/
|
|
)
|
|
assert.match(
|
|
workflow,
|
|
/test "\$\{TARGET_CELL_IDS\}" = "production-gce-c12,production-gce-c13"/
|
|
)
|
|
assert.match(
|
|
workflow,
|
|
/test "\$\{TARGET_CELL_IDS\}" = "production-gce-c7,production-gce-c8,production-gce-c10,production-gce-c13,production-gce-c17,production-gce-c18"/
|
|
)
|
|
const jobGate = workflow.slice(
|
|
workflow.indexOf('jobs:'),
|
|
workflow.indexOf('runs-on:')
|
|
)
|
|
assert.doesNotMatch(jobGate, /PRODUCTION_GCP_RELAY_FENCE_/)
|
|
const brokerStep = workflow.slice(
|
|
workflow.indexOf('- name: Invoke private target-supersession broker'),
|
|
workflow.indexOf('- name: Preflight or run multi-target evacuation')
|
|
)
|
|
assert.match(brokerStep, /steps\.google-fence-broker-auth\.outputs\.id_token/)
|
|
assert.doesNotMatch(brokerStep, /PRODUCTION_GCP_DEPLOY_SERVICE_ACCOUNT/)
|
|
const sourceFenceStep = workflow.slice(
|
|
workflow.indexOf('- name: Invoke private source-fence broker'),
|
|
workflow.indexOf('- name: Preflight or run multi-target evacuation')
|
|
)
|
|
assert.match(sourceFenceStep, /steps\.google-fence-broker-auth\.outputs\.id_token/)
|
|
assert.match(sourceFenceStep, /\/v1\/fence-source/)
|
|
assert.doesNotMatch(sourceFenceStep, /PRODUCTION_GCP_DEPLOY_SERVICE_ACCOUNT/)
|
|
})
|
|
|
|
test('Terraform binds dedicated identities to exact OIDC and resource boundaries', async () => {
|
|
const terraform = await source('infra/terraform/relay-github-actions.tf')
|
|
for (const claim of ['job_workflow_ref', 'workflow_ref', 'ref', 'environment']) {
|
|
assert.match(terraform, new RegExp(`assertion\\.${claim}`))
|
|
}
|
|
assert.match(terraform, /github_monitor_workflow_file/)
|
|
assert.match(terraform, /github_fence_workflow_file/)
|
|
assert.match(terraform, /github_production_relay_capacity_job_workflow_file/)
|
|
assert.match(terraform, /google_service_account" "github_monitor"/)
|
|
assert.match(terraform, /google_service_account" "github_fence"/)
|
|
assert.match(terraform, /google_service_account\.github_monitor\[0\]\.member/)
|
|
assert.match(terraform, /service_account_id = google_service_account\.github_fence\[0\]\.name/)
|
|
assert.match(terraform, /attribute\.relay_ops_identity\/monitor/)
|
|
assert.match(terraform, /attribute\.relay_ops_identity\/fence/)
|
|
assert.doesNotMatch(terraform, /github_relay_fence_operator/)
|
|
assert.doesNotMatch(terraform, /github_terraform_fence_state_writer/)
|
|
const broker = await source('infra/terraform/relay-fence-broker.tf')
|
|
assert.match(broker, /max_instance_request_concurrency = 1/)
|
|
assert.match(broker, /max_instance_count = 1/)
|
|
assert.match(broker, /roles\/run\.invoker/)
|
|
assert.match(broker, /google_service_account\.github_fence\[0\]\.member/)
|
|
assert.doesNotMatch(broker, /allUsers/)
|
|
const brokerDeploy = readRelayWorkflow('deploy-relay-fence-broker.yml')
|
|
assert.match(brokerDeploy, /sha-\$\{GITHUB_SHA\}/)
|
|
assert.match(brokerDeploy, /gcloud run services update/)
|
|
assert.match(brokerDeploy, /\.status\.traffic/)
|
|
assert.doesNotMatch(brokerDeploy, /latestReadyRevisionName/)
|
|
assert.doesNotMatch(brokerDeploy, /--set-env-vars/)
|
|
})
|
|
|
|
test('Terraform exposes the audited production environment values', async () => {
|
|
const outputs = await source('infra/terraform/outputs.tf')
|
|
for (const output of [
|
|
'github_relay_monitor_workload_identity_provider',
|
|
'github_relay_monitor_service_account',
|
|
'github_relay_fence_workload_identity_provider',
|
|
'github_relay_fence_service_account'
|
|
]) {
|
|
assert.match(outputs, new RegExp(`output "${output}"`))
|
|
}
|
|
})
|
|
|
|
test('production mutations pass the minted admin token to live preflight', async () => {
|
|
const workflow = readRelayWorkflow('deploy-relay-production.yml')
|
|
const recheck = workflow.slice(
|
|
workflow.indexOf('- name: Recheck all live safety signals'),
|
|
workflow.indexOf('- name: Create single-use dry-run marker')
|
|
)
|
|
assert.match(
|
|
recheck,
|
|
/ORCA_RELAY_ADMIN_ID_TOKEN: \$\{\{ steps\.google-auth\.outputs\.id_token \}\}/
|
|
)
|
|
const multiTarget = readRelayWorkflow('deploy-relay-production-multi-target.yml')
|
|
const multiTargetRecheck = multiTarget.slice(
|
|
multiTarget.indexOf('- name: Recheck all live safety signals'),
|
|
multiTarget.indexOf('- name: Create single-use dry-run marker')
|
|
)
|
|
assert.match(multiTargetRecheck, /steps\.google-auth\.outputs\.id_token/)
|
|
assert.match(multiTargetRecheck, /inputs\.mode != 'supersede-target'/)
|
|
})
|
|
|
|
test('fence broker pins the production-proven Terraform planner', async () => {
|
|
const dockerfile = await source('apps/relay-fence-broker/Dockerfile')
|
|
assert.match(dockerfile, /FROM hashicorp\/terraform:1\.15\.8 AS terraform/)
|
|
})
|
|
|
|
test('push uses its dedicated identity and rollout lease', () => {
|
|
const workflow = readRelayWorkflow('push-deploy.yml')
|
|
assert.match(workflow, /PRODUCTION_GCP_PUSH_DEPLOY_WORKLOAD_IDENTITY_PROVIDER/)
|
|
assert.match(workflow, /PRODUCTION_GCP_PUSH_DEPLOY_SERVICE_ACCOUNT/)
|
|
assert.doesNotMatch(workflow, /PRODUCTION_GCP_RELAY_DEPLOY_/)
|
|
assert.match(workflow, /group: production-push-rollout/)
|
|
assert.match(workflow, /object: terraform\/state\/push-rollout\/production.lock/)
|
|
})
|