mirror of
https://github.com/stablyai/orca.git
synced 2026-10-30 16:47:34 +00:00
Move Relay deployment configuration into private cloud operations (#31792)
* refactor(cloud): move deployment configuration into private operations repo * docs(cloud): clarify private operator ownership * fix(ci): install pnpm through npm-compatible bootstrap * fix(ci): avoid native downloader metadata timeout * test(push): move deployment preflight contract into private repo * fix(ci): bootstrap Node before installing pnpm in minimal containers * fix(ci): pass Windows boundary test paths through the Node runner * fix(ci): run Windows boundary Vitest with installed Node
This commit is contained in:
Neil
+10
-46
@@ -25,13 +25,9 @@ the validation required before rolling out the updated images and host pins.
|
||||
mutation lease, the Terraform checkout, and the narrow Compute mutation used
|
||||
when a registered target is superseded. The workflow that calls it holds read
|
||||
and invoke rights only, never those mutation permissions.
|
||||
- `apps/relay-ops`: the relay operations console and the incident monitor
|
||||
behind `pnpm ops:relay`, `pnpm incident:relay`, and
|
||||
`pnpm incident:relay-preflight`.
|
||||
- `apps/push` and `packages/push-contract`: the mobile push gateway that holds
|
||||
the APNs key and sends to phones through APNs and FCM, and its wire contract.
|
||||
It is deployed and operated from here but is not part of the relay data path;
|
||||
see [docs/push-gateway.md](docs/push-gateway.md).
|
||||
Deployment and operations are owned by the private `stablyai/orca-cloud` repository.
|
||||
|
||||
## Mobile push gateway
|
||||
|
||||
@@ -58,53 +54,21 @@ and local development. Configure it with `ORCA_PUSH_PUBLIC_URL`, `ORCA_PUSH_FCM_
|
||||
`ORCA_PUSH_APNS_KEY_ID`, `ORCA_PUSH_APPLE_TEAM_ID`, all three or none), and
|
||||
optionally `ORCA_PUSH_APNS_TOPIC`. The FCM credential comes from
|
||||
the runtime service account, so no key material is configured for Android. See
|
||||
[push gateway operations](docs/push-gateway.md) for deployment and recovery.
|
||||
[push gateway operations](https://github.com/stablyai/orca-cloud/blob/main/cloud/docs/push-gateway.md) for deployment and recovery.
|
||||
|
||||
Logging is aggregate counters only. Tokens, notification titles, notification
|
||||
bodies, and full host fingerprints never reach a log line.
|
||||
|
||||
## Infrastructure and operations
|
||||
## Deployment configuration
|
||||
|
||||
- `infra/terraform`: the relay Terraform root. It owns the cells, the director,
|
||||
the shared Cloud SQL instance, DNS, observability, and every GitHub Workload
|
||||
Identity provider the relay workflows authenticate through. `backend/` holds
|
||||
the per-environment backend configuration and `environments/` the tfvars.
|
||||
Drive it through `pnpm infra:init`, `pnpm infra:plan`, and `pnpm infra:apply`.
|
||||
- `dev/scripts`: the deploy, capacity, admission, rehome, monitoring, and load
|
||||
scripts the workflows call, plus the contract tests that pin each workflow
|
||||
and Terraform surface. Run them with `pnpm test`.
|
||||
- `dev/contracts` and `dev/fixtures`: the checked-in data those contract tests
|
||||
read, including the Terraform root partition.
|
||||
- `docs/`: the relay runbooks, capacity-testing guide, incident-monitor
|
||||
reference, the workflow variable reference in `docs/relay-workflows.md`, and
|
||||
the push gateway runbook in `docs/push-gateway.md`.
|
||||
The relay runtime and wire contracts are public. Production and staging Terraform,
|
||||
backend configuration, operator tools, runbooks, broker image assembly, and deployment
|
||||
workflows live in the private `stablyai/orca-cloud` repository. That repository pins
|
||||
an immutable public source revision and combines it with its deployment configuration.
|
||||
|
||||
## Workflows
|
||||
|
||||
The 25 `.github/workflows/cloud-*.yml` workflows are the deploy and operate
|
||||
surface: publish and deploy the director, roll GCE cell capacity, operate Asia
|
||||
admission and regional rehoming, prove staging capacity, monitor production,
|
||||
power staging up and down, and deploy the mobile push gateway.
|
||||
`.github/actions/cloud-sql-rollout-lease` is the compare-and-swap lease that
|
||||
serializes rollouts against the shared Cloud SQL instance. Push reuses that
|
||||
action with its own lease object and deployment concurrency group.
|
||||
|
||||
Every one of them is inert. Each top-level job is gated on
|
||||
`vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true'`, a repository variable that is
|
||||
unset here, so the two scheduled triggers and every manual dispatch skip
|
||||
without running a step. Only the repository owner, holding the GCP identities
|
||||
these workflows authenticate as, can turn them on.
|
||||
|
||||
`Cloud Verify` is not gated. It builds, typechecks, lints, tests, secret-scans,
|
||||
and validates the relay Terraform on every change under `cloud/`, and it runs
|
||||
on fork pull requests, so it configures no backend and holds no credential.
|
||||
|
||||
## What is not here
|
||||
|
||||
The `terraform-foundation` and `terraform-apps` roots and the API and auth
|
||||
services live in the private `stablyai/orca-cloud` repository. Scripts and
|
||||
tests that spanned both trees were narrowed to the relay side rather than
|
||||
carrying a dangling reference.
|
||||
`Cloud Verify` here builds and tests public runtime code, audits dependencies, scans
|
||||
relay and push images, and checks for secrets without cloud credentials. Private CI
|
||||
also validates Terraform, scans the SQL proxy and broker, and runs deployment contracts.
|
||||
|
||||
## Local development
|
||||
|
||||
|
||||
@@ -1,17 +0,0 @@
|
||||
import { readFileSync } from 'node:fs'
|
||||
import { expect, it } from 'vitest'
|
||||
import { loadPushConfig } from './config.js'
|
||||
|
||||
it('runs the deployment image preflight against the actual config loader', () => {
|
||||
const workflow = readFileSync(
|
||||
new URL('../../../../.github/workflows/cloud-push-deploy.yml', import.meta.url),
|
||||
'utf8'
|
||||
)
|
||||
const step = workflow.split('- name: Require image support for inert validation')[1]!
|
||||
const script = step.match(/--input-type=module -e '([\s\S]*?)'/)?.[1]
|
||||
expect(script).toBeDefined()
|
||||
const run = new Function('loadPushConfig', script!.replace(/import .*?;/, ''))
|
||||
expect(() => run(loadPushConfig)).not.toThrow()
|
||||
expect(() => run(() => ({ mode: 'active' }))).toThrow('validation_mode_unsupported')
|
||||
expect(() => run(() => ({ mode: 'validation' }))).toThrow('validation_mode_not_fail_closed')
|
||||
})
|
||||
@@ -1,51 +0,0 @@
|
||||
FROM node:24.22.1-alpine@sha256:feb76650b7195319268c59cb2d419c732d6c499b13bf5296f16ab59eb1f0a4f3 AS build
|
||||
WORKDIR /workspace
|
||||
ENV PATH="/opt/pnpm/node_modules/.bin:$PATH"
|
||||
COPY package.json pnpm-lock.yaml pnpm-workspace.yaml tsconfig.base.json ./
|
||||
RUN npm install --prefix /opt/pnpm --ignore-scripts --no-audit --no-fund "$(node -p 'require("./package.json").packageManager.split("+")[0]')" \
|
||||
&& npm audit signatures --prefix /opt/pnpm
|
||||
COPY apps/relay-fence-broker/package.json apps/relay-fence-broker/package.json
|
||||
RUN pnpm install --frozen-lockfile
|
||||
COPY apps/relay-fence-broker apps/relay-fence-broker
|
||||
RUN pnpm --filter @orca-cloud/relay-fence-broker build
|
||||
|
||||
FROM hashicorp/terraform:1.17.0@sha256:4d1b496a256f2989bccdb04b9104a4ede97234b10fba90259e173e905ec4fb58 AS terraform
|
||||
|
||||
FROM gcr.io/google.com/cloudsdktool/google-cloud-cli:590.0.0-alpine@sha256:2d6788b06641d02cff063b337dd01a66b71b813bd22fb1fa1b9ee2e7c0ef0743
|
||||
RUN apk upgrade --no-cache && apk add --no-cache libstdc++
|
||||
ARG ORCA_RELAY_FENCE_IMAGE_COMMIT
|
||||
RUN test "$(printf '%s' "${ORCA_RELAY_FENCE_IMAGE_COMMIT}" | grep -E '^[a-f0-9]{40}$')"
|
||||
ENV NODE_ENV=production
|
||||
ENV PORT=8080
|
||||
ENV ORCA_RELAY_FENCE_IMAGE_COMMIT=${ORCA_RELAY_FENCE_IMAGE_COMMIT}
|
||||
ENV IAC_TOOL=terraform
|
||||
ENV CLOUDSDK_STORAGE_USE_GCLOUD_CRC32C=false
|
||||
WORKDIR /workspace
|
||||
COPY --from=build /usr/local /usr/local
|
||||
COPY --from=build /opt/pnpm /opt/pnpm
|
||||
ENV PATH="/opt/pnpm/node_modules/.bin:$PATH"
|
||||
COPY --from=terraform /bin/terraform /usr/local/bin/terraform
|
||||
COPY package.json pnpm-lock.yaml pnpm-workspace.yaml ./
|
||||
COPY apps/relay-fence-broker/package.json apps/relay-fence-broker/package.json
|
||||
COPY --from=build /workspace/apps/relay-fence-broker/dist apps/relay-fence-broker/dist
|
||||
COPY dev/scripts dev/scripts
|
||||
COPY infra/terraform infra/terraform
|
||||
COPY apps/relay-fence-broker/requirements.txt /tmp/broker-requirements.txt
|
||||
RUN /google-cloud-sdk/platform/bundledpythonunixmusl/bin/python3 -m pip install \
|
||||
--no-cache-dir --no-deps --require-hashes --upgrade \
|
||||
--target /google-cloud-sdk/lib/third_party -r /tmp/broker-requirements.txt \
|
||||
&& rm -rf /google-cloud-sdk/platform/bq /google-cloud-sdk/platform/gsutil \
|
||||
/google-cloud-sdk/platform/bundledpythonunixmusl/lib/python3.14/site-packages/pip* \
|
||||
/google-cloud-sdk/platform/bundledpythonunixmusl/lib/python3.14/site-packages/wheel* \
|
||||
&& rm -f /google-cloud-sdk/bin/gcloud-crc32c /google-cloud-sdk/bin/bq \
|
||||
/google-cloud-sdk/bin/gsutil /google-cloud-sdk/platform/bundledpythonunixmusl/bin/pip* \
|
||||
/google-cloud-sdk/platform/bundledpythonunixmusl/bin/wheel /tmp/broker-requirements.txt
|
||||
RUN pnpm install --prod --frozen-lockfile --filter @orca-cloud/relay-fence-broker... \
|
||||
&& rm -rf /opt/pnpm /usr/local/lib/node_modules /opt/yarn-* /root/.cache /root/.local \
|
||||
&& rm -f /usr/local/bin/npm /usr/local/bin/npx /usr/local/bin/corepack \
|
||||
/usr/local/bin/pnpm /usr/local/bin/pnpx /usr/local/bin/yarn /usr/local/bin/yarnpkg \
|
||||
&& adduser -D -u 10001 broker \
|
||||
&& chown -R broker:broker /workspace
|
||||
USER broker
|
||||
EXPOSE 8080
|
||||
CMD ["node", "apps/relay-fence-broker/dist/index.js"]
|
||||
@@ -1,66 +0,0 @@
|
||||
# Orca Relay Operations
|
||||
|
||||
A private, aggregate dashboard for the Orca Relay control and data planes. It reads local `gcloud` and `gh` credentials on the server; credentials and per-user Relay state never enter the browser. One cached `gcloud auth print-access-token` refresh feeds concurrent read-only Google APIs so the collector does not stampede the local credential store.
|
||||
|
||||
## Run locally
|
||||
|
||||
Prerequisites:
|
||||
|
||||
- Node 24 and pnpm 10
|
||||
- `gcloud` authenticated for `onorca-cloud` and `onorca-cloud-staging`
|
||||
- `gh` authenticated with read access to `stablyai/orca-cloud`
|
||||
|
||||
From the repository root:
|
||||
|
||||
```sh
|
||||
pnpm install
|
||||
pnpm ops:relay
|
||||
```
|
||||
|
||||
Open <http://127.0.0.1:2455>. The server binds only to loopback and refreshes aggregate data every minute. Production and staging are read-only by default.
|
||||
|
||||
The cost panel is a labeled planning estimate. There is currently no Cloud Billing export in either project, so the dashboard cannot claim exact billed spend. GCP Billing remains authoritative.
|
||||
|
||||
## Share through Tailscale
|
||||
|
||||
Keep the dashboard bound to loopback and let Tailscale provide identity, TLS, and tailnet ACL enforcement:
|
||||
|
||||
```sh
|
||||
tailscale serve --bg http://127.0.0.1:2455
|
||||
tailscale serve status
|
||||
```
|
||||
|
||||
Share the HTTPS URL printed by `tailscale serve status` with the team. Limit access to the intended operator group in the tailnet ACL. Do not use a public funnel. Stop sharing with:
|
||||
|
||||
```sh
|
||||
tailscale serve reset
|
||||
```
|
||||
|
||||
For a persistent host, run `pnpm --filter @orca-cloud/relay-ops build` and supervise `pnpm --filter @orca-cloud/relay-ops start` with the host's normal process manager. The process needs the same non-interactive `gcloud` and `gh` identities.
|
||||
|
||||
## Optional staging controls
|
||||
|
||||
Controls are intentionally local-only and disabled unless explicitly enabled:
|
||||
|
||||
```sh
|
||||
RELAY_OPS_ENABLE_STAGING_CONTROLS=1 pnpm ops:relay
|
||||
```
|
||||
|
||||
Even in this mode the service never changes GCP directly. It dispatches `.github/workflows/power-relay-staging.yml`, preserves the workflow's typed `WAKE_STAGING` / `SLEEP_STAGING` confirmation, and always wakes only configured-admission cells. Requests require the loopback origin and a per-process CSRF token, so controls stay unavailable through the Tailscale view.
|
||||
|
||||
## Data and security boundaries
|
||||
|
||||
- Browser payloads contain aggregate Monitoring points, resource health, immutable image digests, alert-policy metadata, and workflow metadata.
|
||||
- Account IDs, host IDs, device IDs, pairing state, assignment rows, bearer tokens, service-account tokens, startup scripts, secret values, and individual Relay-admin state are excluded.
|
||||
- Sleeping staging is inventory-only. Viewing it does not probe or cold-start Cloud Run services and cannot resize empty MIGs.
|
||||
- Partial GCP or GitHub failures degrade the affected panel and produce a sanitized warning.
|
||||
- Missing cell inventory renders as `Unknown`, never `Sleeping`. After one successful read, transient credential or collector failures retain the last good snapshot and mark it stale.
|
||||
- Responses use `no-store`, a restrictive CSP, frame denial, and no-referrer headers.
|
||||
|
||||
## Verification
|
||||
|
||||
```sh
|
||||
pnpm --filter @orca-cloud/relay-ops test
|
||||
pnpm --filter @orca-cloud/relay-ops typecheck
|
||||
pnpm --filter @orca-cloud/relay-ops build
|
||||
```
|
||||
@@ -1,30 +0,0 @@
|
||||
{
|
||||
"name": "@orca-cloud/relay-ops",
|
||||
"private": true,
|
||||
"version": "0.0.0",
|
||||
"type": "module",
|
||||
"main": "dist/index.js",
|
||||
"scripts": {
|
||||
"build": "pnpm clean && tsc -p tsconfig.build.json && node -e \"require('fs').cpSync('public','dist/public',{recursive:true})\"",
|
||||
"clean": "node -e \"require('fs').rmSync('dist', { recursive: true, force: true })\"",
|
||||
"dev": "tsx watch src/index.ts",
|
||||
"incident:monitor": "tsx src/incident-monitor-cli.ts",
|
||||
"incident:preflight": "tsx src/incident-live-preflight-cli.ts",
|
||||
"incident:pre-drain-sample": "tsx src/pre-drain-sample-cli.ts",
|
||||
"lint": "tsc -p tsconfig.json --noEmit",
|
||||
"start": "node dist/index.js",
|
||||
"test": "vitest run",
|
||||
"typecheck": "tsc -p tsconfig.json --noEmit"
|
||||
},
|
||||
"dependencies": {
|
||||
"@hono/node-server": "^2.1.5",
|
||||
"hono": "^4.13.13",
|
||||
"zod": "^4.6.7"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/node": "^24.19.2",
|
||||
"tsx": "^4.23.15",
|
||||
"typescript": "^7.0.2",
|
||||
"vitest": "^5.0.5"
|
||||
}
|
||||
}
|
||||
@@ -1,273 +0,0 @@
|
||||
const state = { environment: 'production', window: 360, snapshot: null, config: null, loading: false }
|
||||
|
||||
const $ = (selector) => document.querySelector(selector)
|
||||
const all = (selector) => [...document.querySelectorAll(selector)]
|
||||
const escapeHtml = (value) => String(value).replace(/[&<>'"]/g, (character) => ({
|
||||
'&': '&', '<': '<', '>': '>', "'": ''', '"': '"'
|
||||
})[character])
|
||||
|
||||
function formatNumber(value, maximumFractionDigits = 0) {
|
||||
if (value === null || value === undefined) return '—'
|
||||
return new Intl.NumberFormat('en-US', { notation: value >= 10_000 ? 'compact' : 'standard', maximumFractionDigits }).format(value)
|
||||
}
|
||||
|
||||
function formatBytes(value) {
|
||||
if (value === null || value === undefined) return '—'
|
||||
const units = ['B', 'KB', 'MB', 'GB', 'TB']
|
||||
let amount = value
|
||||
let unit = 0
|
||||
while (Math.abs(amount) >= 1000 && unit < units.length - 1) { amount /= 1000; unit += 1 }
|
||||
return `${formatNumber(amount, amount < 10 ? 1 : 0)} ${units[unit]}`
|
||||
}
|
||||
|
||||
function formatMetric(metric, value) {
|
||||
if (value === null || value === undefined) return '—'
|
||||
if (metric.unit === 'bytes') return formatBytes(value)
|
||||
if (metric.unit === 'milliseconds') return `${formatNumber(value, 1)} ms`
|
||||
return formatNumber(value, value < 10 ? 1 : 0)
|
||||
}
|
||||
|
||||
function timeAgo(value) {
|
||||
const seconds = Math.max(0, Math.round((Date.now() - Date.parse(value)) / 1000))
|
||||
if (seconds < 60) return `${seconds}s ago`
|
||||
if (seconds < 3600) return `${Math.floor(seconds / 60)}m ago`
|
||||
if (seconds < 86400) return `${Math.floor(seconds / 3600)}h ago`
|
||||
return `${Math.floor(seconds / 86400)}d ago`
|
||||
}
|
||||
|
||||
function shortImage(value) {
|
||||
const digest = value?.match(/sha256:([a-f0-9]{64})/)?.[1]
|
||||
if (digest) return digest.slice(0, 10)
|
||||
const tag = value?.split(':').at(-1)
|
||||
return tag?.slice(0, 16) ?? '—'
|
||||
}
|
||||
|
||||
function badge(label, healthy) {
|
||||
return `<span class="badge ${healthy === true ? 'healthy' : healthy === false ? 'unhealthy' : ''}">${escapeHtml(label)}</span>`
|
||||
}
|
||||
|
||||
function renderSummary(snapshot) {
|
||||
const { summary } = snapshot
|
||||
const utilization = summary.poweredCapacity === null
|
||||
? null
|
||||
: summary.poweredCapacity > 0
|
||||
? summary.observedConnections / summary.poweredCapacity * 100
|
||||
: 0
|
||||
const items = [
|
||||
['Observed connections', formatNumber(summary.observedConnections, 1), 'Latest 1-minute aggregate mean'],
|
||||
['Active relay sessions', formatNumber(summary.observedSplices, 1), `${formatNumber(summary.observedControls, 1)} desktop-control mean`],
|
||||
['Healthy cells', `${summary.activeCells ?? '—'} / ${summary.totalCells}`, summary.poweredCapacity === null ? 'Cell inventory unavailable' : `${formatNumber(summary.poweredCapacity)} powered request units`],
|
||||
['Capacity signal', utilization === null ? '—' : `${formatNumber(utilization, 2)}%`, `${formatNumber(summary.configuredCapacity)} configured admission units`]
|
||||
]
|
||||
$('#summary').innerHTML = items.map(([label, value, caption]) => `
|
||||
<article class="metric-card">
|
||||
<p class="eyebrow">${escapeHtml(label)}</p>
|
||||
<div class="value">${escapeHtml(value)}</div>
|
||||
<div class="caption">${escapeHtml(caption)}</div>
|
||||
</article>`).join('')
|
||||
}
|
||||
|
||||
function cellState(cell) {
|
||||
if (cell.targetSize === null) return ['Unknown', null]
|
||||
if (cell.targetSize === 0) return ['Sleeping', null]
|
||||
if (cell.backendHealth === 'healthy' && cell.endpoint.ready) return ['Healthy', true]
|
||||
if (cell.stable && cell.backendHealth === 'empty') return ['Starting', null]
|
||||
return ['Attention', false]
|
||||
}
|
||||
|
||||
function renderTopology(snapshot) {
|
||||
const director = snapshot.resources.director
|
||||
const sleeping = snapshot.resources.cells.every((cell) => cell.targetSize === 0)
|
||||
&& snapshot.resources.sql?.activationPolicy === 'NEVER'
|
||||
const directorHealthy = director?.ready && snapshot.resources.directorEndpoint.health
|
||||
$('#topology-meta').textContent = `${snapshot.environment.project} · ${snapshot.environment.region}`
|
||||
const cellNodes = snapshot.resources.cells.map((cell) => {
|
||||
const [label, healthy] = cellState(cell)
|
||||
return `<div class="topology-node">
|
||||
<strong>${escapeHtml(cell.hostname.toUpperCase())} ${badge(label, healthy)}</strong>
|
||||
<span>${escapeHtml(cell.region)} · ${escapeHtml(cell.zone)} · ${formatNumber(cell.capacityRequests)} units</span>
|
||||
</div>`
|
||||
}).join('')
|
||||
$('#topology').innerHTML = `
|
||||
<div class="topology-node"><strong>Desktop + phone</strong><span>Encrypted Relay traffic</span></div>
|
||||
<div class="connector" aria-hidden="true"></div>
|
||||
<div class="topology-node"><strong>Director ${badge(sleeping ? 'Sleeping' : directorHealthy ? 'Ready' : 'Attention', sleeping ? null : Boolean(directorHealthy))}</strong><span>${escapeHtml(snapshot.environment.directorOrigin)}</span></div>
|
||||
<div class="connector" aria-hidden="true"></div>
|
||||
<div class="topology-cells">${cellNodes}</div>`
|
||||
}
|
||||
|
||||
function chartPaths(points, width = 400, height = 112) {
|
||||
if (points.length === 0) return null
|
||||
const values = points.map((point) => point.value)
|
||||
const minimum = Math.min(0, ...values)
|
||||
const maximum = Math.max(...values)
|
||||
const spread = maximum - minimum || 1
|
||||
const coordinates = points.map((point, index) => {
|
||||
const x = points.length === 1 ? width : index / (points.length - 1) * width
|
||||
const y = height - ((point.value - minimum) / spread * (height - 10) + 5)
|
||||
return [x, y]
|
||||
})
|
||||
const line = coordinates.map(([x, y], index) => `${index === 0 ? 'M' : 'L'}${x.toFixed(1)},${y.toFixed(1)}`).join(' ')
|
||||
const area = `${line} L${width},${height} L0,${height} Z`
|
||||
return { line, area }
|
||||
}
|
||||
|
||||
function renderCharts(snapshot) {
|
||||
const names = ['controls', 'splices', 'assignment_5xx', 'postgres_retries', 'auth_failures', 'event_loop_ms_p99']
|
||||
$('#charts').innerHTML = names.map((name) => {
|
||||
const metric = snapshot.monitoring.metrics[name]
|
||||
const paths = chartPaths(metric.points)
|
||||
const graph = paths ? `<svg class="chart" viewBox="0 0 400 112" preserveAspectRatio="none" role="img" aria-label="${escapeHtml(metric.label)} over time">
|
||||
<line x1="0" y1="111" x2="400" y2="111"></line>
|
||||
<path class="area" d="${paths.area}"></path><path class="line" d="${paths.line}"></path>
|
||||
</svg>` : '<div class="chart-empty">No samples in this window</div>'
|
||||
return `<article class="panel chart-card"><div class="chart-head"><div><p class="eyebrow">${escapeHtml(metric.label)}</p><div class="chart-value">${escapeHtml(formatMetric(metric, metric.latest))}</div></div><span class="chart-unit">${escapeHtml(metric.unit)}</span></div>${graph}</article>`
|
||||
}).join('')
|
||||
}
|
||||
|
||||
function renderCells(snapshot) {
|
||||
const controlsByCell = snapshot.monitoring.metrics.controls.latestByCell
|
||||
const splicesByCell = snapshot.monitoring.metrics.splices.latestByCell
|
||||
$('#cells').innerHTML = snapshot.resources.cells.map((cell) => {
|
||||
const [label, healthy] = cellState(cell)
|
||||
const observed = (controlsByCell[cell.cellId] ?? 0) + (splicesByCell[cell.cellId] ?? 0)
|
||||
return `<tr>
|
||||
<td><strong>${escapeHtml(cell.hostname.toUpperCase())}</strong><div class="row-caption">${escapeHtml(cell.region)} · ${escapeHtml(cell.zone)}</div></td>
|
||||
<td>${badge(label, healthy)}<div class="row-caption">MIG ${cell.runningInstances ?? '—'}/${cell.targetSize ?? '—'}</div></td>
|
||||
<td>${formatNumber(observed, 1)}<div class="row-caption">1-minute mean</div></td>
|
||||
<td>${formatNumber(cell.capacityRequests)}<div class="row-caption">DB pool ${formatNumber(cell.databasePoolMax)} · ${cell.configuredAdmission ? 'configured' : 'candidate only'}</div></td>
|
||||
<td class="mono">${escapeHtml(shortImage(cell.imageDigest))}</td>
|
||||
</tr>`
|
||||
}).join('')
|
||||
}
|
||||
|
||||
function serviceRow(name, healthy, detail, suffix = '') {
|
||||
return `<div class="service-row"><div><div class="row-title">${escapeHtml(name)}</div><div class="row-caption">${escapeHtml(detail)}</div></div>${badge(suffix || (healthy ? 'Ready' : 'Attention'), healthy)}</div>`
|
||||
}
|
||||
|
||||
function renderServices(snapshot) {
|
||||
const { resources } = snapshot
|
||||
const sleeping = resources.cells.every((cell) => cell.targetSize === 0)
|
||||
&& resources.sql?.activationPolicy === 'NEVER'
|
||||
const certDays = resources.certificate?.expireTime
|
||||
? Math.floor((Date.parse(resources.certificate.expireTime) - Date.now()) / 86400000)
|
||||
: null
|
||||
$('#services').innerHTML = [
|
||||
serviceRow('Director', sleeping ? null : Boolean(resources.director?.ready && resources.directorEndpoint.health), `${resources.director?.revision ?? 'Revision unavailable'} · ${shortImage(resources.director?.image)}`, sleeping ? 'Sleeping' : ''),
|
||||
serviceRow('Authentication', sleeping ? null : Boolean(resources.auth?.ready && resources.authEndpoint.health), `${resources.auth?.revision ?? 'Revision unavailable'} · ${shortImage(resources.auth?.image)}`, sleeping ? 'Sleeping' : ''),
|
||||
serviceRow('Cloud SQL', resources.sql?.state === 'RUNNABLE' || resources.sql?.activationPolicy === 'NEVER', `${resources.sql?.tier ?? 'unknown'} · ${resources.sql?.activationPolicy ?? 'unknown'}`, resources.sql?.state ?? 'Unknown'),
|
||||
serviceRow('Wildcard TLS', resources.certificate?.state === 'ACTIVE', resources.certificate?.domains.join(', ') ?? 'Certificate unavailable', certDays === null ? 'Unknown' : `${certDays}d`)
|
||||
].join('')
|
||||
}
|
||||
|
||||
function renderAlerts(snapshot) {
|
||||
const policies = snapshot.monitoring.alertPolicies
|
||||
$('#alerts').innerHTML = policies.length ? policies.map((policy) => `<div class="list-row"><div><div class="row-title">${escapeHtml(policy.displayName.replace('Orca Relay: ', ''))}</div><div class="row-caption">Cloud Monitoring policy</div></div>${badge(policy.enabled ? 'Enabled' : 'Disabled', policy.enabled)}</div>`).join('') : '<p class="muted">No Relay alert policies returned.</p>'
|
||||
}
|
||||
|
||||
function renderWorkflows(snapshot) {
|
||||
$('#workflows').innerHTML = snapshot.workflows.length ? snapshot.workflows.slice(0, 6).map((run) => {
|
||||
const healthy = run.conclusion === 'success'
|
||||
const stateLabel = run.status === 'completed' ? (run.conclusion ?? 'completed') : run.status
|
||||
return `<a class="list-row" href="${escapeHtml(run.url)}" target="_blank" rel="noreferrer"><div><div class="row-title">${escapeHtml(run.name)}</div><div class="row-caption">${escapeHtml(run.headSha)} · ${timeAgo(run.updatedAt)}</div></div>${badge(stateLabel, healthy)}</a>`
|
||||
}).join('') : '<p class="muted">Workflow history unavailable.</p>'
|
||||
}
|
||||
|
||||
function renderCost(snapshot) {
|
||||
const cost = snapshot.cost
|
||||
$('#cost').innerHTML = `<div class="cost-total"><strong>$${formatNumber(cost.monthlyUsd)}</strong><span class="muted">/ month</span></div>
|
||||
<p class="row-caption">Modeled range $${formatNumber(cost.rangeUsd[0])}–$${formatNumber(cost.rangeUsd[1])}. Not billed cost.</p>
|
||||
<div class="cost-lines">${cost.lines.map((line) => `<div class="cost-line"><span>${escapeHtml(line.label)}</span><strong>$${formatNumber(line.monthlyUsd, 2)}</strong></div>`).join('')}</div>
|
||||
<p class="cost-note"><strong>Exact billing:</strong> ${escapeHtml(cost.actualBilling.reason)}<br>${escapeHtml(cost.caveats[1])}</p>`
|
||||
}
|
||||
|
||||
function renderWarnings(snapshot) {
|
||||
const warning = $('#warnings')
|
||||
warning.classList.toggle('hidden', snapshot.warnings.length === 0)
|
||||
warning.textContent = snapshot.warnings.length ? `Partial data: ${snapshot.warnings.join(' ')}` : ''
|
||||
}
|
||||
|
||||
function render(snapshot) {
|
||||
state.snapshot = snapshot
|
||||
$('#environment-label').textContent = `${snapshot.environment.label} · ${snapshot.environment.region}`
|
||||
$('#freshness').textContent = snapshot.stale
|
||||
? `Last good update ${timeAgo(snapshot.generatedAt)} · refresh degraded`
|
||||
: `Updated ${timeAgo(snapshot.generatedAt)}`
|
||||
$('#freshness-dot').className = snapshot.stale ? 'status-dot neutral' : 'status-dot healthy'
|
||||
renderWarnings(snapshot)
|
||||
renderSummary(snapshot)
|
||||
renderTopology(snapshot)
|
||||
renderCharts(snapshot)
|
||||
renderCells(snapshot)
|
||||
renderServices(snapshot)
|
||||
renderAlerts(snapshot)
|
||||
renderWorkflows(snapshot)
|
||||
renderCost(snapshot)
|
||||
$('#power-panel').classList.toggle('hidden', !(state.environment === 'staging' && state.config?.stagingControlsEnabled))
|
||||
$('#compute-link').href = snapshot.environment.consoleLinks.compute
|
||||
$('#alert-link').href = snapshot.environment.consoleLinks.alerts
|
||||
}
|
||||
|
||||
async function loadSnapshot() {
|
||||
if (state.loading) return
|
||||
state.loading = true
|
||||
$('#refresh').disabled = true
|
||||
$('#error').classList.add('hidden')
|
||||
$('#freshness-dot').className = 'status-dot neutral'
|
||||
$('#freshness').textContent = 'Refreshing current state…'
|
||||
try {
|
||||
const response = await fetch(`/api/snapshot?environment=${state.environment}&window=${state.window}`)
|
||||
const body = await response.json()
|
||||
if (!response.ok) throw new Error(body.error ?? 'Snapshot failed')
|
||||
render(body)
|
||||
} catch (error) {
|
||||
$('#freshness-dot').className = 'status-dot unhealthy'
|
||||
$('#freshness').textContent = 'Refresh failed'
|
||||
$('#error').textContent = error instanceof Error ? error.message : 'Relay operations data is unavailable.'
|
||||
$('#error').classList.remove('hidden')
|
||||
} finally {
|
||||
state.loading = false
|
||||
$('#refresh').disabled = false
|
||||
}
|
||||
}
|
||||
|
||||
async function dispatchPower(mode) {
|
||||
const confirmation = mode === 'wake' ? 'WAKE_STAGING' : mode === 'sleep' ? 'SLEEP_STAGING' : ''
|
||||
if (confirmation) {
|
||||
const entered = window.prompt(`Type ${confirmation} to dispatch the guarded workflow.`) ?? ''
|
||||
if (entered !== confirmation) return
|
||||
}
|
||||
all('[data-power]').forEach((button) => { button.disabled = true })
|
||||
$('#power-result').textContent = 'Dispatching…'
|
||||
try {
|
||||
const response = await fetch('/api/staging/power', {
|
||||
method: 'POST',
|
||||
headers: { 'content-type': 'application/json', 'x-csrf-token': state.config.csrfToken },
|
||||
body: JSON.stringify({ mode, confirmation })
|
||||
})
|
||||
const body = await response.json()
|
||||
if (!response.ok) throw new Error(body.error)
|
||||
$('#power-result').textContent = 'Workflow accepted. Refresh after it completes.'
|
||||
} catch (error) {
|
||||
$('#power-result').textContent = error instanceof Error ? error.message : 'Dispatch failed.'
|
||||
} finally {
|
||||
all('[data-power]').forEach((button) => { button.disabled = false })
|
||||
}
|
||||
}
|
||||
|
||||
all('[data-environment]').forEach((button) => button.addEventListener('click', () => {
|
||||
state.environment = button.dataset.environment
|
||||
all('[data-environment]').forEach((candidate) => candidate.setAttribute('aria-pressed', String(candidate === button)))
|
||||
loadSnapshot()
|
||||
}))
|
||||
$('#window').addEventListener('change', (event) => { state.window = Number(event.target.value); loadSnapshot() })
|
||||
$('#refresh').addEventListener('click', loadSnapshot)
|
||||
all('[data-power]').forEach((button) => button.addEventListener('click', () => dispatchPower(button.dataset.power)))
|
||||
|
||||
async function start() {
|
||||
try { state.config = await fetch('/api/config').then((response) => response.json()) } catch { state.config = {} }
|
||||
await loadSnapshot()
|
||||
window.setInterval(loadSnapshot, 60_000)
|
||||
}
|
||||
|
||||
start()
|
||||
@@ -1,115 +0,0 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<meta name="color-scheme" content="light dark" />
|
||||
<title>Orca Relay Operations</title>
|
||||
<link rel="icon" href="data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 32 32'%3E%3Crect width='32' height='32' rx='9' fill='%23171717'/%3E%3Ctext x='16' y='22' text-anchor='middle' font-family='sans-serif' font-weight='700' font-size='17' fill='white'%3EO%3C/text%3E%3C/svg%3E" />
|
||||
<link rel="stylesheet" href="/styles.css" />
|
||||
<script type="module" src="/app.js"></script>
|
||||
</head>
|
||||
<body>
|
||||
<header class="topbar">
|
||||
<div class="brand">
|
||||
<span class="brand-mark" aria-hidden="true">O</span>
|
||||
<div>
|
||||
<strong>Orca Relay</strong>
|
||||
<span>Operations</span>
|
||||
</div>
|
||||
</div>
|
||||
<div class="toolbar">
|
||||
<div class="segmented" aria-label="Environment">
|
||||
<button type="button" data-environment="production" aria-pressed="true">Production</button>
|
||||
<button type="button" data-environment="staging" aria-pressed="false">Staging</button>
|
||||
</div>
|
||||
<select id="window" aria-label="Metrics window">
|
||||
<option value="60">Last hour</option>
|
||||
<option value="360" selected>Last 6 hours</option>
|
||||
<option value="1440">Last 24 hours</option>
|
||||
</select>
|
||||
<button type="button" id="refresh" class="button secondary">Refresh</button>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
<main>
|
||||
<section class="page-heading">
|
||||
<div>
|
||||
<p class="eyebrow" id="environment-label">Production · us-central1</p>
|
||||
<h1>Relay data plane</h1>
|
||||
<p class="lede">Private, aggregate operations view. No account, host, device, or pairing identifiers.</p>
|
||||
</div>
|
||||
<div class="freshness">
|
||||
<span class="status-dot neutral" id="freshness-dot"></span>
|
||||
<span id="freshness">Loading current state…</span>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<div id="error" class="banner error hidden" role="alert"></div>
|
||||
<div id="warnings" class="banner warning hidden"></div>
|
||||
|
||||
<section class="summary-grid" id="summary" aria-label="Relay summary">
|
||||
<article class="metric-card skeleton"></article>
|
||||
<article class="metric-card skeleton"></article>
|
||||
<article class="metric-card skeleton"></article>
|
||||
<article class="metric-card skeleton"></article>
|
||||
</section>
|
||||
|
||||
<section class="panel topology-panel">
|
||||
<div class="panel-heading">
|
||||
<div><p class="eyebrow">Topology</p><h2>Request path</h2></div>
|
||||
<span class="meta" id="topology-meta"></span>
|
||||
</div>
|
||||
<div class="topology" id="topology"></div>
|
||||
</section>
|
||||
|
||||
<section class="section-heading"><div><p class="eyebrow">Traffic</p><h2>Signals in selected window</h2></div></section>
|
||||
<section class="chart-grid" id="charts"></section>
|
||||
|
||||
<section class="two-column">
|
||||
<article class="panel">
|
||||
<div class="panel-heading"><div><p class="eyebrow">Compute</p><h2>Relay cells</h2></div><a class="meta panel-link" id="compute-link" target="_blank" rel="noreferrer">Open in GCP ↗</a></div>
|
||||
<p class="panel-note">Configured admission is shown below. Live director admission and heartbeat state stays unavailable because this dashboard has no Relay-admin identity.</p>
|
||||
<div class="table-wrap"><table>
|
||||
<thead><tr><th>Cell</th><th>State</th><th>Observed</th><th>Capacity</th><th>Image</th></tr></thead>
|
||||
<tbody id="cells"></tbody>
|
||||
</table></div>
|
||||
</article>
|
||||
<article class="panel">
|
||||
<div class="panel-heading"><div><p class="eyebrow">Control plane</p><h2>Services</h2></div></div>
|
||||
<div class="service-list" id="services"></div>
|
||||
</article>
|
||||
</section>
|
||||
|
||||
<section class="three-column">
|
||||
<article class="panel">
|
||||
<div class="panel-heading"><div><p class="eyebrow">Guardrails</p><h2>Alert policies</h2></div><a class="meta panel-link" id="alert-link" target="_blank" rel="noreferrer">Open incidents ↗</a></div>
|
||||
<div class="list" id="alerts"></div>
|
||||
</article>
|
||||
<article class="panel">
|
||||
<div class="panel-heading"><div><p class="eyebrow">Delivery</p><h2>Recent workflows</h2></div></div>
|
||||
<div class="list" id="workflows"></div>
|
||||
</article>
|
||||
<article class="panel">
|
||||
<div class="panel-heading"><div><p class="eyebrow">Planning</p><h2>Monthly run-rate</h2></div></div>
|
||||
<div id="cost"></div>
|
||||
</article>
|
||||
</section>
|
||||
|
||||
<section class="panel hidden" id="power-panel">
|
||||
<div class="panel-heading">
|
||||
<div><p class="eyebrow">Staging only</p><h2>Power workflow</h2></div>
|
||||
<span class="badge">GitHub guarded</span>
|
||||
</div>
|
||||
<p class="muted">Dispatches the reviewed GitHub workflow. It never changes GCP resources directly from this process.</p>
|
||||
<div class="power-actions">
|
||||
<button class="button secondary" type="button" data-power="status">Check status</button>
|
||||
<button class="button secondary" type="button" data-power="wake">Wake configured cells</button>
|
||||
<button class="button secondary" type="button" data-power="sleep">Sleep staging</button>
|
||||
<span id="power-result" class="meta"></span>
|
||||
</div>
|
||||
</section>
|
||||
</main>
|
||||
<footer>Orca Relay Operations · Server-side credentials · Read-only by default</footer>
|
||||
</body>
|
||||
</html>
|
||||
@@ -1,165 +0,0 @@
|
||||
:root {
|
||||
--radius: 10px;
|
||||
--background: #fff;
|
||||
--foreground: #0a0a0a;
|
||||
--card: #fff;
|
||||
--primary: #171717;
|
||||
--primary-foreground: #fafafa;
|
||||
--secondary: #f5f5f5;
|
||||
--muted: #f5f5f5;
|
||||
--muted-foreground: #737373;
|
||||
--accent: #f5f5f5;
|
||||
--border: #e5e5e5;
|
||||
--ring: #a1a1a1;
|
||||
--destructive: #e40014;
|
||||
--success: #15803d;
|
||||
--warning: #895503;
|
||||
--font-mono: 'SF Mono', SFMono-Regular, ui-monospace, Menlo, Consolas, monospace;
|
||||
}
|
||||
@media (prefers-color-scheme: dark) {
|
||||
:root {
|
||||
--background: #0a0a0a;
|
||||
--foreground: #fafafa;
|
||||
--card: #171717;
|
||||
--primary: #e5e5e5;
|
||||
--primary-foreground: #171717;
|
||||
--secondary: #262626;
|
||||
--muted: #262626;
|
||||
--muted-foreground: #a1a1a1;
|
||||
--accent: #262626;
|
||||
--border: rgb(255 255 255 / 0.07);
|
||||
--ring: #737373;
|
||||
--destructive: #ff6568;
|
||||
--success: #4ade80;
|
||||
--warning: #fbbf24;
|
||||
}
|
||||
}
|
||||
* { box-sizing: border-box; }
|
||||
body {
|
||||
margin: 0;
|
||||
background: var(--background);
|
||||
color: var(--foreground);
|
||||
font-family: 'Geist', -apple-system, BlinkMacSystemFont, 'Segoe UI', sans-serif;
|
||||
font-size: 14px;
|
||||
letter-spacing: .01em;
|
||||
}
|
||||
button, select { font: inherit; }
|
||||
button:focus-visible, select:focus-visible { outline: 2px solid var(--ring); outline-offset: 2px; }
|
||||
.topbar {
|
||||
height: 64px;
|
||||
border-bottom: 1px solid var(--border);
|
||||
padding: 0 max(24px, calc((100vw - 1440px) / 2));
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
position: sticky;
|
||||
top: 0;
|
||||
background: color-mix(in srgb, var(--background) 92%, transparent);
|
||||
backdrop-filter: blur(16px);
|
||||
z-index: 10;
|
||||
}
|
||||
.brand { display: flex; align-items: center; gap: 10px; }
|
||||
.brand-mark {
|
||||
width: 30px; height: 30px; border-radius: 9px; background: var(--primary);
|
||||
color: var(--primary-foreground); display: grid; place-items: center; font-weight: 700;
|
||||
}
|
||||
.brand div { display: flex; align-items: baseline; gap: 7px; }
|
||||
.brand span:last-child { color: var(--muted-foreground); font-size: 12px; }
|
||||
.toolbar { display: flex; align-items: center; gap: 8px; }
|
||||
.segmented { display: flex; padding: 3px; gap: 2px; border-radius: 8px; background: var(--secondary); }
|
||||
.segmented button { border: 0; background: transparent; color: var(--muted-foreground); padding: 5px 10px; border-radius: 6px; cursor: pointer; }
|
||||
.segmented button[aria-pressed="true"] { background: var(--card); color: var(--foreground); box-shadow: 0 1px 2px rgb(0 0 0 / .08); }
|
||||
select, .button { height: 32px; border: 1px solid var(--border); border-radius: 8px; background: var(--card); color: var(--foreground); padding: 0 10px; }
|
||||
.button { cursor: pointer; font-weight: 500; }
|
||||
.button:hover { background: var(--accent); }
|
||||
.button:disabled { opacity: .5; cursor: wait; }
|
||||
main { max-width: 1440px; margin: 0 auto; padding: 42px 24px 72px; }
|
||||
.page-heading { display: flex; justify-content: space-between; align-items: end; gap: 24px; margin-bottom: 28px; }
|
||||
h1, h2, p { margin: 0; }
|
||||
h1 { font-size: 28px; line-height: 1.2; letter-spacing: -.025em; margin-top: 5px; }
|
||||
h2 { font-size: 16px; line-height: 1.25; letter-spacing: -.01em; margin-top: 3px; }
|
||||
.lede { color: var(--muted-foreground); margin-top: 8px; max-width: 680px; }
|
||||
.eyebrow { color: var(--muted-foreground); font-size: 11px; line-height: 1; font-weight: 600; text-transform: uppercase; letter-spacing: .05em; }
|
||||
.freshness { color: var(--muted-foreground); display: flex; align-items: center; gap: 8px; font-size: 12px; white-space: nowrap; }
|
||||
.status-dot { width: 7px; height: 7px; border-radius: 999px; background: var(--ring); display: inline-block; }
|
||||
.status-dot.healthy { background: var(--success); }
|
||||
.status-dot.unhealthy { background: var(--destructive); }
|
||||
.summary-grid { display: grid; grid-template-columns: repeat(4, minmax(0, 1fr)); gap: 12px; margin-bottom: 12px; }
|
||||
.metric-card, .panel { border: 1px solid var(--border); border-radius: 14px; background: var(--card); }
|
||||
.metric-card { padding: 18px; min-height: 116px; }
|
||||
.metric-card .value { font-size: 29px; font-weight: 600; letter-spacing: -.035em; margin-top: 16px; }
|
||||
.metric-card .caption { color: var(--muted-foreground); font-size: 12px; margin-top: 3px; }
|
||||
.skeleton { background: linear-gradient(90deg, var(--card), var(--muted), var(--card)); background-size: 200% 100%; animation: shimmer 1.6s infinite; }
|
||||
@keyframes shimmer { to { background-position: -200% 0; } }
|
||||
.panel { padding: 18px; }
|
||||
.topology-panel { margin-bottom: 36px; }
|
||||
.panel-heading, .section-heading { display: flex; align-items: center; justify-content: space-between; gap: 16px; margin-bottom: 18px; }
|
||||
.section-heading { margin-top: 34px; }
|
||||
.topology { display: grid; grid-template-columns: 1fr 48px 1fr 48px 2fr; align-items: stretch; gap: 8px; }
|
||||
.topology-node { border: 1px solid var(--border); background: var(--background); border-radius: 10px; padding: 14px; min-width: 0; }
|
||||
.topology-node strong { display: block; font-size: 13px; }
|
||||
.topology-node span { display: block; color: var(--muted-foreground); font-size: 12px; margin-top: 4px; overflow: hidden; text-overflow: ellipsis; }
|
||||
.topology-cells { display: grid; grid-template-columns: repeat(3, minmax(0, 1fr)); gap: 8px; }
|
||||
.connector { display: grid; place-items: center; color: var(--muted-foreground); }
|
||||
.connector::before { content: ''; width: 100%; height: 1px; background: var(--border); }
|
||||
.chart-grid { display: grid; grid-template-columns: repeat(3, minmax(0, 1fr)); gap: 12px; margin-bottom: 36px; }
|
||||
.chart-card { min-height: 220px; }
|
||||
.chart-head { display: flex; justify-content: space-between; align-items: start; }
|
||||
.chart-value { font-size: 22px; font-weight: 600; letter-spacing: -.03em; }
|
||||
.chart-unit { color: var(--muted-foreground); font-size: 11px; }
|
||||
.chart { width: 100%; height: 122px; margin-top: 18px; overflow: visible; }
|
||||
.chart path.area { fill: color-mix(in srgb, var(--foreground) 5%, transparent); }
|
||||
.chart path.line { fill: none; stroke: var(--foreground); stroke-width: 1.5; vector-effect: non-scaling-stroke; }
|
||||
.chart line { stroke: var(--border); stroke-width: 1; }
|
||||
.chart-empty { color: var(--muted-foreground); height: 120px; display: grid; place-items: center; font-size: 12px; }
|
||||
.two-column { display: grid; grid-template-columns: 1.7fr 1fr; gap: 12px; margin-bottom: 12px; }
|
||||
.three-column { display: grid; grid-template-columns: repeat(3, minmax(0, 1fr)); gap: 12px; margin-bottom: 12px; }
|
||||
.two-column > *, .three-column > *, .chart-grid > * { min-width: 0; }
|
||||
.table-wrap { overflow-x: auto; }
|
||||
table { width: 100%; border-collapse: collapse; font-size: 12px; }
|
||||
th { color: var(--muted-foreground); font-weight: 500; text-align: left; padding: 0 10px 10px; }
|
||||
td { padding: 12px 10px; border-top: 1px solid var(--border); vertical-align: middle; }
|
||||
td:first-child, th:first-child { padding-left: 0; }
|
||||
td:last-child, th:last-child { padding-right: 0; }
|
||||
.mono { font-family: var(--font-mono); font-size: 11px; }
|
||||
.badge { display: inline-flex; align-items: center; border: 1px solid var(--border); border-radius: 999px; padding: 2px 7px; font-size: 11px; color: var(--muted-foreground); }
|
||||
.badge.healthy { color: var(--success); border-color: color-mix(in srgb, var(--success) 25%, var(--border)); }
|
||||
.badge.unhealthy { color: var(--destructive); border-color: color-mix(in srgb, var(--destructive) 25%, var(--border)); }
|
||||
.service-list, .list { display: grid; }
|
||||
.service-row, .list-row { padding: 11px 0; border-top: 1px solid var(--border); display: flex; align-items: center; justify-content: space-between; gap: 12px; min-width: 0; }
|
||||
.service-row:first-child, .list-row:first-child { border-top: 0; padding-top: 0; }
|
||||
.service-row:last-child, .list-row:last-child { padding-bottom: 0; }
|
||||
.row-title { font-size: 12px; font-weight: 500; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
|
||||
.row-caption, .meta, .muted { color: var(--muted-foreground); font-size: 11px; }
|
||||
.panel-note { color: var(--muted-foreground); font-size: 11px; line-height: 1.45; margin: -8px 0 14px; }
|
||||
.panel-link:hover { color: var(--foreground); text-decoration: underline; }
|
||||
.row-caption { margin-top: 3px; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
|
||||
a { color: inherit; text-decoration: none; }
|
||||
a:hover .row-title { text-decoration: underline; }
|
||||
.cost-total { display: flex; align-items: baseline; gap: 7px; margin-bottom: 12px; }
|
||||
.cost-total strong { font-size: 28px; letter-spacing: -.035em; }
|
||||
.cost-line { display: flex; justify-content: space-between; gap: 10px; padding: 7px 0; border-top: 1px solid var(--border); font-size: 12px; }
|
||||
.cost-note { color: var(--muted-foreground); font-size: 11px; line-height: 1.5; margin-top: 12px; }
|
||||
.banner { border: 1px solid var(--border); border-radius: 10px; padding: 11px 13px; margin-bottom: 12px; font-size: 12px; }
|
||||
.banner.error { color: var(--destructive); }
|
||||
.banner.warning { color: var(--warning); }
|
||||
.hidden { display: none !important; }
|
||||
.power-actions { display: flex; align-items: center; gap: 8px; margin-top: 14px; flex-wrap: wrap; }
|
||||
footer { border-top: 1px solid var(--border); padding: 24px; color: var(--muted-foreground); font-size: 11px; text-align: center; }
|
||||
@media (max-width: 1050px) {
|
||||
.summary-grid, .chart-grid { grid-template-columns: repeat(2, minmax(0, 1fr)); }
|
||||
.three-column { grid-template-columns: 1fr; }
|
||||
.topology { grid-template-columns: 1fr; }
|
||||
.connector { height: 20px; }
|
||||
.connector::before { height: 100%; width: 1px; }
|
||||
}
|
||||
@media (max-width: 760px) {
|
||||
.topbar { height: auto; min-height: 64px; padding: 12px 16px; align-items: stretch; flex-direction: column; gap: 12px; }
|
||||
.brand div { display: grid; gap: 0; }
|
||||
.toolbar { flex-wrap: wrap; justify-content: flex-start; }
|
||||
main { padding: 28px 16px 56px; }
|
||||
.page-heading { align-items: flex-start; flex-direction: column; }
|
||||
.summary-grid, .chart-grid, .two-column { grid-template-columns: 1fr; }
|
||||
.topology-cells { grid-template-columns: 1fr; }
|
||||
table { min-width: 600px; }
|
||||
}
|
||||
@@ -1,85 +0,0 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { buildCostModel } from './cost-model.js'
|
||||
import {
|
||||
RELAY_OPS_ENVIRONMENTS,
|
||||
relayOpsCellsFromTerraform,
|
||||
type RelayOpsCellConfig
|
||||
} from './environment-config.js'
|
||||
import type { ResourceInventory } from './resource-inventory.js'
|
||||
|
||||
const regionalCellsSource = `
|
||||
relay_gce_cells = {
|
||||
"staging-gce-c3" = {
|
||||
hostname = "c3"
|
||||
zone = "us-central1-a"
|
||||
machine_type = "e2-standard-2"
|
||||
capacity_requests = 4000
|
||||
}
|
||||
"staging-gce-c4" = {
|
||||
hostname = "c4"
|
||||
region = "asia-east2"
|
||||
zone = "asia-east2-a"
|
||||
machine_type = "e2-standard-4"
|
||||
capacity_requests = 6000
|
||||
database_pool_max = 10
|
||||
initially_enabled = false
|
||||
}
|
||||
}
|
||||
`
|
||||
|
||||
function inventory(
|
||||
targetSize: number,
|
||||
activationPolicy: string,
|
||||
cells: RelayOpsCellConfig[] = RELAY_OPS_ENVIRONMENTS.staging.cells
|
||||
): ResourceInventory {
|
||||
return {
|
||||
director: null,
|
||||
auth: null,
|
||||
sql: { state: targetSize ? 'RUNNABLE' : 'STOPPED', activationPolicy, tier: 'db-custom-1-3840', availabilityType: 'ZONAL', databaseVersion: 'POSTGRES_17' },
|
||||
certificate: null,
|
||||
directorEndpoint: { health: null, ready: null, latencyMs: null },
|
||||
authEndpoint: { health: null, ready: null, latencyMs: null },
|
||||
cells: cells.map((cell) => ({
|
||||
...cell, migName: `mig-${cell.hostname}`, targetSize, runningInstances: targetSize,
|
||||
stable: true, template: 'template', imageDigest: null,
|
||||
backendHealth: targetSize ? 'healthy' : 'empty',
|
||||
endpoint: { health: null, ready: null, latencyMs: null }
|
||||
})),
|
||||
warnings: []
|
||||
}
|
||||
}
|
||||
|
||||
describe('buildCostModel', () => {
|
||||
it('shows the sleeping staging floor without VM or SQL compute', () => {
|
||||
const result = buildCostModel(RELAY_OPS_ENVIRONMENTS.staging, inventory(0, 'NEVER'))
|
||||
expect(result.kind).toBe('planning-estimate')
|
||||
expect(result.monthlyUsd).toBe(34)
|
||||
expect(result.lines.find((line) => line.label === 'Relay cell VMs')?.monthlyUsd).toBe(0)
|
||||
expect(result.actualBilling.available).toBe(false)
|
||||
expect(result.caveats[0]).toContain('not the Cloud Billing invoice')
|
||||
})
|
||||
|
||||
it('prices durable machine inventory and network floors by region', () => {
|
||||
const cells = relayOpsCellsFromTerraform({
|
||||
environment: 'staging',
|
||||
domain: 'relay-staging.onorca.dev',
|
||||
source: regionalCellsSource
|
||||
})
|
||||
const environment = { ...RELAY_OPS_ENVIRONMENTS.staging, cells }
|
||||
const result = buildCostModel(environment, inventory(1, 'NEVER', cells))
|
||||
const machines = result.lines.find((line) => line.label === 'Relay cell VMs')
|
||||
const network = result.lines.find((line) => line.label === 'Load balancer and network floor')
|
||||
|
||||
expect(machines).toEqual({
|
||||
label: 'Relay cell VMs',
|
||||
monthlyUsd: 185.79,
|
||||
basis: '2 configured VM cells at regional machine rates × 730 hours'
|
||||
})
|
||||
|
||||
expect(network).toEqual({
|
||||
label: 'Load balancer and network floor',
|
||||
monthlyUsd: 29,
|
||||
basis: 'shared HTTPS foundation plus NAT floor in 2 configured regions'
|
||||
})
|
||||
})
|
||||
})
|
||||
@@ -1,115 +0,0 @@
|
||||
import type {
|
||||
RelayOpsEnvironment,
|
||||
RelayOpsMachineType,
|
||||
RelayOpsRegion
|
||||
} from './environment-config.js'
|
||||
import type { ResourceInventory } from './resource-inventory.js'
|
||||
|
||||
export type CostLine = {
|
||||
label: string
|
||||
monthlyUsd: number
|
||||
basis: string
|
||||
}
|
||||
|
||||
export type CostModel = {
|
||||
kind: 'planning-estimate'
|
||||
monthlyUsd: number
|
||||
rangeUsd: [number, number]
|
||||
actualBilling: { available: false; reason: string }
|
||||
lines: CostLine[]
|
||||
caveats: string[]
|
||||
}
|
||||
|
||||
const HOURS_PER_MONTH = 730
|
||||
const MACHINE_HOURLY_USD: Record<
|
||||
RelayOpsRegion,
|
||||
Record<RelayOpsMachineType, number>
|
||||
> = {
|
||||
'us-central1': {
|
||||
'e2-standard-2': 0.06701142,
|
||||
'e2-standard-4': 0.13402284
|
||||
},
|
||||
'asia-east2': {
|
||||
'e2-standard-2': 0.0938,
|
||||
'e2-standard-4': 0.1875
|
||||
}
|
||||
}
|
||||
const SHARED_NETWORK_FLOOR_USD = 19
|
||||
const REGIONAL_NAT_FLOOR_USD = 5
|
||||
|
||||
function round(value: number): number {
|
||||
return Math.round(value * 100) / 100
|
||||
}
|
||||
|
||||
export function buildCostModel(
|
||||
environment: RelayOpsEnvironment,
|
||||
resources: ResourceInventory
|
||||
): CostModel {
|
||||
const runningCells = resources.cells.filter((cell) => (cell.targetSize ?? 0) > 0)
|
||||
const runningCellCount = runningCells.reduce((sum, cell) => sum + (cell.targetSize ?? 0), 0)
|
||||
const compute = runningCells.reduce(
|
||||
(sum, cell) =>
|
||||
sum + (cell.targetSize ?? 0) * MACHINE_HOURLY_USD[cell.region][cell.machineType],
|
||||
0
|
||||
) * HOURS_PER_MONTH
|
||||
const disks = runningCellCount * 30 * 0.1
|
||||
const sqlRunning = resources.sql?.activationPolicy === 'ALWAYS'
|
||||
const sql = sqlRunning ? (environment.id === 'production' ? 105 : 52) : 0
|
||||
const cloudRunMinimums =
|
||||
(resources.director?.minInstances ?? 0) + (resources.auth?.minInstances ?? 0)
|
||||
const cloudRun = cloudRunMinimums * 10
|
||||
const configuredRegions = new Set(
|
||||
(resources.cells.length > 0 ? resources.cells : environment.cells).map((cell) => cell.region)
|
||||
)
|
||||
const networkFoundation =
|
||||
SHARED_NETWORK_FLOOR_USD + configuredRegions.size * REGIONAL_NAT_FLOOR_USD
|
||||
const observability = environment.id === 'production' ? 12 : 5
|
||||
const lines: CostLine[] = [
|
||||
{
|
||||
label: 'Relay cell VMs',
|
||||
monthlyUsd: round(compute),
|
||||
basis: `${runningCellCount} configured VM cells at regional machine rates × 730 hours`
|
||||
},
|
||||
{
|
||||
label: 'Cell boot disks',
|
||||
monthlyUsd: round(disks),
|
||||
basis: `${runningCellCount} × 30 GB balanced persistent disk`
|
||||
},
|
||||
{
|
||||
label: 'Cloud SQL',
|
||||
monthlyUsd: sql,
|
||||
basis: sqlRunning ? `${resources.sql?.tier ?? 'configured tier'} active` : 'stopped'
|
||||
},
|
||||
{
|
||||
label: 'Cloud Run minimums',
|
||||
monthlyUsd: cloudRun,
|
||||
basis: `${cloudRunMinimums} configured minimum instances`
|
||||
},
|
||||
{
|
||||
label: 'Load balancer and network floor',
|
||||
monthlyUsd: networkFoundation,
|
||||
basis: `shared HTTPS foundation plus NAT floor in ${configuredRegions.size} configured region${configuredRegions.size === 1 ? '' : 's'}`
|
||||
},
|
||||
{
|
||||
label: 'Logs and monitoring allowance',
|
||||
monthlyUsd: observability,
|
||||
basis: 'planning allowance; varies with traffic and retention'
|
||||
}
|
||||
]
|
||||
const monthlyUsd = round(lines.reduce((sum, line) => sum + line.monthlyUsd, 0))
|
||||
return {
|
||||
kind: 'planning-estimate',
|
||||
monthlyUsd,
|
||||
rangeUsd: [round(monthlyUsd * 0.85), round(monthlyUsd * 1.3)],
|
||||
actualBilling: {
|
||||
available: false,
|
||||
reason: 'Cloud Billing export is not configured in either Relay project.'
|
||||
},
|
||||
lines,
|
||||
caveats: [
|
||||
'This is a modeled run-rate, not the Cloud Billing invoice.',
|
||||
'Network egress, actual request traffic, credits, discounts, taxes, and free tiers are excluded.',
|
||||
'Use the GCP Billing report for authoritative spend and forecasts.'
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -1,62 +0,0 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { DashboardSnapshotCache } from './dashboard-snapshot.js'
|
||||
import type { DashboardSnapshot } from './dashboard-snapshot.js'
|
||||
import type { GcloudClient } from './gcloud-client.js'
|
||||
|
||||
const gcloud: GcloudClient = { accessToken: async () => 'a'.repeat(40) }
|
||||
|
||||
function snapshot(kind: 'good' | 'unavailable'): DashboardSnapshot {
|
||||
const good = kind === 'good'
|
||||
return {
|
||||
generatedAt: '2026-07-15T12:00:00.000Z',
|
||||
resources: {
|
||||
director: good ? {} : null,
|
||||
auth: good ? {} : null,
|
||||
sql: good ? {} : null,
|
||||
cells: [{ targetSize: good ? 1 : null }]
|
||||
},
|
||||
monitoring: {
|
||||
warnings: good
|
||||
? []
|
||||
: ['Cloud Monitoring credentials are unavailable. Run gcloud auth login.']
|
||||
},
|
||||
summary: { observedConnections: good ? 7 : 0 },
|
||||
warnings: good ? [] : ['Google Cloud credentials are unavailable. Run gcloud auth login.'],
|
||||
stale: false,
|
||||
staleReason: null
|
||||
} as unknown as DashboardSnapshot
|
||||
}
|
||||
|
||||
describe('DashboardSnapshotCache', () => {
|
||||
it('keeps the last good view when a later credential refresh fails', async () => {
|
||||
let calls = 0
|
||||
const cache = new DashboardSnapshotCache(gcloud, 0, async () => {
|
||||
calls += 1
|
||||
return snapshot(calls === 1 ? 'good' : 'unavailable')
|
||||
})
|
||||
|
||||
const first = await cache.read('production', 30)
|
||||
const second = await cache.read('production', 31)
|
||||
|
||||
expect(first.stale).toBe(false)
|
||||
expect(second.stale).toBe(true)
|
||||
expect(second.summary.observedConnections).toBe(7)
|
||||
expect(second.staleReason).toContain('credentials')
|
||||
})
|
||||
|
||||
it('keeps the last good view when a later collector throws', async () => {
|
||||
let calls = 0
|
||||
const cache = new DashboardSnapshotCache(gcloud, 0, async () => {
|
||||
calls += 1
|
||||
if (calls > 1) throw new Error('sensitive collector context')
|
||||
return snapshot('good')
|
||||
})
|
||||
|
||||
await cache.read('production', 30)
|
||||
const second = await cache.read('production', 31)
|
||||
|
||||
expect(second.stale).toBe(true)
|
||||
expect(second.summary.observedConnections).toBe(7)
|
||||
expect(JSON.stringify(second)).not.toContain('sensitive collector context')
|
||||
})
|
||||
})
|
||||
@@ -1,167 +0,0 @@
|
||||
import type { RelayOpsEnvironmentId } from './environment-config.js'
|
||||
import { relayOpsEnvironment } from './environment-config.js'
|
||||
import type { GcloudClient } from './gcloud-client.js'
|
||||
import { readRelayWorkflowRuns } from './github-runs.js'
|
||||
import { readMonitoringSnapshot } from './monitoring-snapshot.js'
|
||||
import { readResourceInventory } from './resource-inventory.js'
|
||||
import { buildCostModel } from './cost-model.js'
|
||||
|
||||
export type DashboardSnapshot = Awaited<ReturnType<typeof buildDashboardSnapshot>>
|
||||
|
||||
export async function buildDashboardSnapshot(
|
||||
environmentId: RelayOpsEnvironmentId,
|
||||
gcloud: GcloudClient,
|
||||
options: { windowMinutes?: number; fetchImpl?: typeof fetch; now?: Date } = {}
|
||||
) {
|
||||
const generatedAt = (options.now ?? new Date()).toISOString()
|
||||
const environment = relayOpsEnvironment(environmentId)
|
||||
const [monitoringResult, resourceResult, workflowResult] = await Promise.allSettled([
|
||||
readMonitoringSnapshot(environment, gcloud, {
|
||||
...(options.now === undefined ? {} : { now: options.now }),
|
||||
...(options.windowMinutes === undefined ? {} : { windowMinutes: options.windowMinutes }),
|
||||
...(options.fetchImpl === undefined ? {} : { fetchImpl: options.fetchImpl })
|
||||
}),
|
||||
readResourceInventory(environment, gcloud, options.fetchImpl),
|
||||
readRelayWorkflowRuns()
|
||||
])
|
||||
if (monitoringResult.status === 'rejected' || resourceResult.status === 'rejected') {
|
||||
const failed = [
|
||||
monitoringResult.status === 'rejected' ? 'Monitoring snapshot' : null,
|
||||
resourceResult.status === 'rejected' ? 'Resource inventory' : null
|
||||
].filter(Boolean)
|
||||
throw new Error(`${failed.join(' and ')} unavailable`)
|
||||
}
|
||||
const resources = resourceResult.value
|
||||
const monitoring = monitoringResult.value
|
||||
const warnings = [...resources.warnings, ...monitoring.warnings]
|
||||
const poweredDigests = new Set(
|
||||
resources.cells.filter((cell) => (cell.targetSize ?? 0) > 0).map((cell) => cell.imageDigest)
|
||||
)
|
||||
if (poweredDigests.has(null)) warnings.push('A powered cell image digest is unavailable.')
|
||||
if (poweredDigests.size > 1) warnings.push('Powered cells are not serving one immutable digest.')
|
||||
const expectedCertificateDomain = `*.${new URL(environment.cells[0]!.origin).hostname
|
||||
.split('.').slice(1).join('.')}`
|
||||
if (resources.certificate && !resources.certificate.domains.includes(expectedCertificateDomain)) {
|
||||
warnings.push('The Relay certificate domain does not match the configured cell domain.')
|
||||
}
|
||||
if (workflowResult.status === 'rejected') warnings.push('GitHub workflow history is unavailable.')
|
||||
const observedConnections = monitoring.metrics.total_connections.latest ?? 0
|
||||
const observedControls = monitoring.metrics.controls.latest ?? 0
|
||||
const observedSplices = monitoring.metrics.splices.latest ?? 0
|
||||
const configuredCapacity = environment.cells
|
||||
.filter((cell) => cell.configuredAdmission)
|
||||
.reduce((sum, cell) => sum + cell.capacityRequests, 0)
|
||||
const cellInventoryAvailable = resources.cells.every((cell) => cell.targetSize !== null)
|
||||
const poweredCapacity = cellInventoryAvailable
|
||||
? resources.cells
|
||||
.filter((cell) => (cell.targetSize ?? 0) > 0)
|
||||
.reduce((sum, cell) => sum + cell.capacityRequests, 0)
|
||||
: null
|
||||
return {
|
||||
schemaVersion: 1,
|
||||
generatedAt,
|
||||
environment: {
|
||||
id: environment.id,
|
||||
label: environment.label,
|
||||
project: environment.project,
|
||||
region: environment.region,
|
||||
directorOrigin: environment.directorOrigin,
|
||||
authOrigin: environment.authOrigin,
|
||||
consoleLinks: {
|
||||
project: `https://console.cloud.google.com/home/dashboard?project=${environment.project}`,
|
||||
alerts: `https://console.cloud.google.com/monitoring/alerting?project=${environment.project}`,
|
||||
compute: `https://console.cloud.google.com/compute/instanceGroups/list?project=${environment.project}`
|
||||
}
|
||||
},
|
||||
summary: {
|
||||
observedConnections,
|
||||
observedControls,
|
||||
observedSplices,
|
||||
configuredCapacity,
|
||||
poweredCapacity,
|
||||
activeCells: cellInventoryAvailable
|
||||
? resources.cells.filter(
|
||||
(cell) => (cell.targetSize ?? 0) > 0 && cell.backendHealth === 'healthy'
|
||||
).length
|
||||
: null,
|
||||
totalCells: resources.cells.length
|
||||
},
|
||||
resources,
|
||||
monitoring,
|
||||
workflows: workflowResult.status === 'fulfilled' ? workflowResult.value : [],
|
||||
cost: buildCostModel(environment, resources),
|
||||
warnings,
|
||||
stale: false,
|
||||
staleReason: null as string | null
|
||||
}
|
||||
}
|
||||
|
||||
type SnapshotCacheEntry = { snapshot: DashboardSnapshot; expiresAt: number }
|
||||
type SnapshotBuilder = (
|
||||
environment: RelayOpsEnvironmentId,
|
||||
gcloud: GcloudClient,
|
||||
options: { windowMinutes?: number }
|
||||
) => Promise<DashboardSnapshot>
|
||||
|
||||
export class DashboardSnapshotCache {
|
||||
private readonly entries = new Map<string, SnapshotCacheEntry>()
|
||||
private readonly pending = new Map<string, Promise<DashboardSnapshot>>()
|
||||
private readonly lastGood = new Map<RelayOpsEnvironmentId, DashboardSnapshot>()
|
||||
|
||||
constructor(
|
||||
private readonly gcloud: GcloudClient,
|
||||
private readonly ttlMs = 30_000,
|
||||
private readonly builder: SnapshotBuilder = buildDashboardSnapshot
|
||||
) {}
|
||||
|
||||
async read(environment: RelayOpsEnvironmentId, windowMinutes: number): Promise<DashboardSnapshot> {
|
||||
const key = `${environment}:${windowMinutes}`
|
||||
const cached = this.entries.get(key)
|
||||
if (cached && cached.expiresAt > Date.now()) return cached.snapshot
|
||||
const existing = this.pending.get(key)
|
||||
if (existing) return await existing
|
||||
const request = this.builder(environment, this.gcloud, { windowMinutes })
|
||||
.then((snapshot) => {
|
||||
const coreInventoryUnavailable =
|
||||
snapshot.resources.director === null &&
|
||||
snapshot.resources.auth === null &&
|
||||
snapshot.resources.sql === null &&
|
||||
snapshot.resources.cells.every((cell) => cell.targetSize === null)
|
||||
const monitoringCredentialsUnavailable = snapshot.monitoring.warnings.some(
|
||||
(warning) => warning.includes('credentials are unavailable')
|
||||
)
|
||||
const lastGood = this.lastGood.get(environment)
|
||||
const result = (coreInventoryUnavailable || monitoringCredentialsUnavailable) && lastGood
|
||||
? {
|
||||
...lastGood,
|
||||
stale: true,
|
||||
staleReason: 'Local Google Cloud credentials are temporarily unavailable.',
|
||||
warnings: [...new Set([...lastGood.warnings, ...snapshot.warnings])]
|
||||
}
|
||||
: snapshot
|
||||
if (!coreInventoryUnavailable && !monitoringCredentialsUnavailable) {
|
||||
this.lastGood.set(environment, snapshot)
|
||||
}
|
||||
this.entries.set(key, { snapshot: result, expiresAt: Date.now() + this.ttlMs })
|
||||
return result
|
||||
})
|
||||
.catch((error: unknown) => {
|
||||
const lastGood = this.lastGood.get(environment)
|
||||
if (!lastGood) throw error
|
||||
const stale = {
|
||||
...lastGood,
|
||||
stale: true,
|
||||
staleReason: 'The latest operations refresh failed; showing the last good snapshot.',
|
||||
warnings: [...new Set([
|
||||
...lastGood.warnings,
|
||||
'The latest operations refresh failed before a complete snapshot was available.'
|
||||
])]
|
||||
}
|
||||
this.entries.set(key, { snapshot: stale, expiresAt: Date.now() + this.ttlMs })
|
||||
return stale
|
||||
})
|
||||
.finally(() => this.pending.delete(key))
|
||||
this.pending.set(key, request)
|
||||
return await request
|
||||
}
|
||||
}
|
||||
@@ -1,156 +0,0 @@
|
||||
import { readFileSync } from 'node:fs'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import {
|
||||
RELAY_OPS_ENVIRONMENTS,
|
||||
relayOpsCellsFromTerraform
|
||||
} from './environment-config.js'
|
||||
|
||||
const durableUsCell = `
|
||||
"production-gce-c26" = {
|
||||
hostname = "c26"
|
||||
zone = "us-central1-a"
|
||||
machine_type = "e2-standard-4"
|
||||
capacity_requests = 4000
|
||||
initially_enabled = false
|
||||
}
|
||||
`
|
||||
|
||||
const durableAsiaCells = `
|
||||
"production-gce-c27" = {
|
||||
hostname = "c27"
|
||||
region = "asia-east2"
|
||||
zone = "asia-east2-a"
|
||||
machine_type = "e2-standard-4"
|
||||
capacity_requests = 6000
|
||||
database_pool_max = 10
|
||||
initially_enabled = false
|
||||
}
|
||||
"production-gce-c28" = {
|
||||
hostname = "c28"
|
||||
region = "asia-east2"
|
||||
zone = "asia-east2-b"
|
||||
machine_type = "e2-standard-4"
|
||||
capacity_requests = 6000
|
||||
database_pool_max = 10
|
||||
initially_enabled = false
|
||||
}
|
||||
"production-gce-c29" = {
|
||||
hostname = "c29"
|
||||
region = "asia-east2"
|
||||
zone = "asia-east2-c"
|
||||
machine_type = "e2-standard-4"
|
||||
capacity_requests = 6000
|
||||
database_pool_max = 10
|
||||
initially_enabled = false
|
||||
}
|
||||
`
|
||||
|
||||
const durableAsiaSource = `
|
||||
relay_gce_cells = {${durableUsCell}${durableAsiaCells}}
|
||||
`
|
||||
|
||||
const durableUsOnlySource = `
|
||||
relay_gce_cells = {${durableUsCell}}
|
||||
`
|
||||
|
||||
// Why: relay-ops sat at 18 cells for four days after C19-C22 shipped, which threw
|
||||
// `selector membership must contain every configured cell exactly once` and blocked
|
||||
// every production mutation. Reading Terraform here makes that drift fail the build.
|
||||
function terraformCells(environment: 'production' | 'staging'): Array<{
|
||||
cellId: string
|
||||
region: string
|
||||
zone: string
|
||||
machineType: string
|
||||
capacityRequests: number
|
||||
databasePoolMax: number
|
||||
connectionHardCap: number | null
|
||||
}> {
|
||||
const tfvars = readFileSync(
|
||||
fileURLToPath(new URL(`../../../infra/terraform/environments/${environment}.tfvars`, import.meta.url)),
|
||||
'utf8'
|
||||
)
|
||||
const block = /relay_gce_cells\s*=\s*\{([\s\S]*)\n\}/.exec(tfvars)?.[1] ?? ''
|
||||
return [...block.matchAll(/"([a-z]+-gce-c\d+)"\s*=\s*\{([\s\S]*?)\n {2}\}/g)]
|
||||
.map((match) => ({
|
||||
cellId: match[1] ?? '',
|
||||
region: /region\s*=\s*"([^"]+)"/.exec(match[2] ?? '')?.[1] ?? 'us-central1',
|
||||
zone: /zone\s*=\s*"([^"]+)"/.exec(match[2] ?? '')?.[1] ?? '',
|
||||
machineType: /machine_type\s*=\s*"([^"]+)"/.exec(match[2] ?? '')?.[1] ?? '',
|
||||
capacityRequests: Number(/capacity_requests\s*=\s*(\d+)/.exec(match[2] ?? '')?.[1]),
|
||||
databasePoolMax: Number(/database_pool_max\s*=\s*(\d+)/.exec(match[2] ?? '')?.[1] ?? 10),
|
||||
connectionHardCap: ((raw) => (raw === undefined ? null : Number(raw)))(
|
||||
/connection_hard_cap\s*=\s*(\d+)/.exec(match[2] ?? '')?.[1]
|
||||
)
|
||||
}))
|
||||
.sort((left, right) => cellOrdinal(left.cellId) - cellOrdinal(right.cellId))
|
||||
}
|
||||
|
||||
const cellOrdinal = (cellId: string): number => Number(/c(\d+)$/.exec(cellId)?.[1] ?? 0)
|
||||
|
||||
describe('relay operations environment config', () => {
|
||||
it.each(['production', 'staging'] as const)(
|
||||
'matches the %s cells Terraform actually provisions',
|
||||
(environment) => {
|
||||
const expected = terraformCells(environment)
|
||||
expect(expected.length).toBeGreaterThan(0)
|
||||
expect(
|
||||
RELAY_OPS_ENVIRONMENTS[environment].cells.map((cell) => ({
|
||||
cellId: cell.cellId,
|
||||
region: cell.region,
|
||||
zone: cell.zone,
|
||||
machineType: cell.machineType,
|
||||
capacityRequests: cell.capacityRequests,
|
||||
databasePoolMax: cell.databasePoolMax,
|
||||
connectionHardCap: cell.connectionHardCap
|
||||
}))
|
||||
).toEqual(expected)
|
||||
}
|
||||
)
|
||||
|
||||
it('derives hostname and origin from the cell ordinal', () => {
|
||||
const cells = RELAY_OPS_ENVIRONMENTS.production.cells
|
||||
expect(cells.at(-1)).toMatchObject({
|
||||
hostname: `c${cells.length}`,
|
||||
origin: `https://c${cells.length}.relay.onorca.dev`
|
||||
})
|
||||
})
|
||||
|
||||
it('inventories Asia cells only when they exist in durable Terraform', () => {
|
||||
const cells = relayOpsCellsFromTerraform({
|
||||
environment: 'production',
|
||||
domain: 'relay.onorca.dev',
|
||||
source: durableAsiaSource
|
||||
})
|
||||
|
||||
expect(cells.slice(1)).toEqual([
|
||||
{
|
||||
cellId: 'production-gce-c27', hostname: 'c27', origin: 'https://c27.relay.onorca.dev',
|
||||
region: 'asia-east2', zone: 'asia-east2-a', machineType: 'e2-standard-4',
|
||||
capacityRequests: 6000, databasePoolMax: 10, connectionHardCap: null,
|
||||
configuredAdmission: false
|
||||
},
|
||||
{
|
||||
cellId: 'production-gce-c28', hostname: 'c28', origin: 'https://c28.relay.onorca.dev',
|
||||
region: 'asia-east2', zone: 'asia-east2-b', machineType: 'e2-standard-4',
|
||||
capacityRequests: 6000, databasePoolMax: 10, connectionHardCap: null,
|
||||
configuredAdmission: false
|
||||
},
|
||||
{
|
||||
cellId: 'production-gce-c29', hostname: 'c29', origin: 'https://c29.relay.onorca.dev',
|
||||
region: 'asia-east2', zone: 'asia-east2-c', machineType: 'e2-standard-4',
|
||||
capacityRequests: 6000, databasePoolMax: 10, connectionHardCap: null,
|
||||
configuredAdmission: false
|
||||
}
|
||||
])
|
||||
|
||||
const usOnlyCells = relayOpsCellsFromTerraform({
|
||||
environment: 'production',
|
||||
domain: 'relay.onorca.dev',
|
||||
source: durableUsOnlySource
|
||||
})
|
||||
expect(usOnlyCells).toHaveLength(1)
|
||||
expect(usOnlyCells.some((cell) => cell.region === 'asia-east2')).toBe(false)
|
||||
expect(usOnlyCells.some((cell) => cell.cellId === 'production-gce-c27')).toBe(false)
|
||||
})
|
||||
})
|
||||
@@ -1,130 +0,0 @@
|
||||
import { readFileSync } from 'node:fs'
|
||||
import { z } from 'zod'
|
||||
|
||||
export type RelayOpsEnvironmentId = 'production' | 'staging'
|
||||
export type RelayOpsRegion = 'us-central1' | 'asia-east2'
|
||||
export type RelayOpsMachineType = 'e2-standard-2' | 'e2-standard-4'
|
||||
|
||||
export type RelayOpsCellConfig = {
|
||||
cellId: string
|
||||
hostname: string
|
||||
origin: string
|
||||
region: RelayOpsRegion
|
||||
zone: string
|
||||
machineType: RelayOpsMachineType
|
||||
capacityRequests: number
|
||||
databasePoolMax: number
|
||||
// The cell's rendered connectionHardCap; null when Terraform leaves it unset.
|
||||
connectionHardCap: number | null
|
||||
configuredAdmission: boolean
|
||||
}
|
||||
|
||||
export type RelayOpsEnvironment = {
|
||||
id: RelayOpsEnvironmentId
|
||||
label: string
|
||||
project: string
|
||||
region: string
|
||||
directorOrigin: string
|
||||
authOrigin: string
|
||||
directorService: string
|
||||
authService: string
|
||||
sqlInstance: string
|
||||
migPrefix: string
|
||||
certificateName: string
|
||||
cells: RelayOpsCellConfig[]
|
||||
}
|
||||
|
||||
const RegionSchema = z.enum(['us-central1', 'asia-east2'])
|
||||
const MachineTypeSchema = z.enum(['e2-standard-2', 'e2-standard-4'])
|
||||
const EnvironmentSchema = z.enum(['production', 'staging'])
|
||||
|
||||
function cellOrdinal(cellId: string): number {
|
||||
return Number(/c(\d+)$/.exec(cellId)?.[1] ?? 0)
|
||||
}
|
||||
|
||||
function required(body: string, pattern: RegExp, label: string): string {
|
||||
const value = pattern.exec(body)?.[1]
|
||||
if (!value) throw new Error(`Relay Ops could not read ${label} from durable Terraform config`)
|
||||
return value
|
||||
}
|
||||
|
||||
export function relayOpsCellsFromTerraform(input: {
|
||||
environment: RelayOpsEnvironmentId
|
||||
domain: string
|
||||
source: string
|
||||
}): RelayOpsCellConfig[] {
|
||||
const block = /relay_gce_cells\s*=\s*\{([\s\S]*)\n\}/.exec(input.source)?.[1]
|
||||
if (!block) throw new Error('Relay Ops could not read durable Relay cells')
|
||||
return [...block.matchAll(/"([a-z]+-gce-c\d+)"\s*=\s*\{([\s\S]*?)\n {2}\}/g)]
|
||||
.map((match) => {
|
||||
const cellId = match[1]!
|
||||
const body = match[2]!
|
||||
const hostname = required(body, /\bhostname\s*=\s*"([^"]+)"/, `${cellId} hostname`)
|
||||
const configuredAdmission = /\binitially_enabled\s*=\s*(true|false)/.exec(body)?.[1]
|
||||
return {
|
||||
cellId,
|
||||
hostname,
|
||||
origin: `https://${hostname}.${input.domain}`,
|
||||
region: RegionSchema.parse(
|
||||
/\bregion\s*=\s*"([^"]+)"/.exec(body)?.[1] ?? 'us-central1'
|
||||
),
|
||||
zone: required(body, /\bzone\s*=\s*"([^"]+)"/, `${cellId} zone`),
|
||||
machineType: MachineTypeSchema.parse(
|
||||
required(body, /\bmachine_type\s*=\s*"([^"]+)"/, `${cellId} machine type`)
|
||||
),
|
||||
capacityRequests: Number(
|
||||
required(body, /\bcapacity_requests\s*=\s*(\d+)/, `${cellId} capacity`)
|
||||
),
|
||||
databasePoolMax: Number(/\bdatabase_pool_max\s*=\s*(\d+)/.exec(body)?.[1] ?? 10),
|
||||
connectionHardCap: ((raw) => (raw === undefined ? null : Number(raw)))(
|
||||
/\bconnection_hard_cap\s*=\s*(\d+)/.exec(body)?.[1]
|
||||
),
|
||||
configuredAdmission: configuredAdmission === undefined || configuredAdmission === 'true'
|
||||
}
|
||||
})
|
||||
.sort((left, right) => cellOrdinal(left.cellId) - cellOrdinal(right.cellId))
|
||||
}
|
||||
|
||||
function durableCells(environment: RelayOpsEnvironmentId, domain: string): RelayOpsCellConfig[] {
|
||||
const source = readFileSync(
|
||||
// Repository root; infra/terraform moves with this tree, so the relative depth holds.
|
||||
new URL(`../../../infra/terraform/environments/${environment}.tfvars`, import.meta.url),
|
||||
'utf8'
|
||||
)
|
||||
return relayOpsCellsFromTerraform({ environment, domain, source })
|
||||
}
|
||||
|
||||
export const RELAY_OPS_ENVIRONMENTS: Record<RelayOpsEnvironmentId, RelayOpsEnvironment> = {
|
||||
production: {
|
||||
id: 'production',
|
||||
label: 'Production',
|
||||
project: 'onorca-cloud',
|
||||
region: 'us-central1',
|
||||
directorOrigin: 'https://relay.onorca.dev',
|
||||
authOrigin: 'https://login.onorca.dev',
|
||||
directorService: 'orca-cloud-relay',
|
||||
authService: 'orca-cloud-auth',
|
||||
sqlInstance: 'orca-cloud-auth-db',
|
||||
migPrefix: 'orca-cloud-relay-gce-',
|
||||
certificateName: 'orca-cloud-relay-gce',
|
||||
cells: durableCells('production', 'relay.onorca.dev')
|
||||
},
|
||||
staging: {
|
||||
id: 'staging',
|
||||
label: 'Staging',
|
||||
project: 'onorca-cloud-staging',
|
||||
region: 'us-central1',
|
||||
directorOrigin: 'https://relay-staging.onorca.dev',
|
||||
authOrigin: 'https://auth-staging.onorca.dev',
|
||||
directorService: 'orca-cloud-relay-staging',
|
||||
authService: 'orca-cloud-auth-staging',
|
||||
sqlInstance: 'orca-cloud-staging-auth-db',
|
||||
migPrefix: 'orca-cloud-staging-relay-gce-',
|
||||
certificateName: 'orca-cloud-staging-relay-gce',
|
||||
cells: durableCells('staging', 'relay-staging.onorca.dev')
|
||||
}
|
||||
}
|
||||
|
||||
export function relayOpsEnvironment(value: unknown): RelayOpsEnvironment {
|
||||
return RELAY_OPS_ENVIRONMENTS[EnvironmentSchema.parse(value)]
|
||||
}
|
||||
@@ -1,43 +0,0 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { createGcloudClient } from './gcloud-client.js'
|
||||
|
||||
describe('createGcloudClient', () => {
|
||||
it('shares and caches one credential refresh across concurrent readers', async () => {
|
||||
let calls = 0
|
||||
const token = 'a'.repeat(40)
|
||||
const client = createGcloudClient(async () => {
|
||||
calls += 1
|
||||
await Promise.resolve()
|
||||
return token
|
||||
})
|
||||
|
||||
const values = await Promise.all([
|
||||
client.accessToken(),
|
||||
client.accessToken(),
|
||||
client.accessToken()
|
||||
])
|
||||
|
||||
expect(values).toEqual([token, token, token])
|
||||
expect(await client.accessToken()).toBe(token)
|
||||
expect(calls).toBe(1)
|
||||
})
|
||||
|
||||
it('caches bounded identity tokens by audience', async () => {
|
||||
const commands: string[][] = []
|
||||
const token = 'aaa.bbb.ccc'
|
||||
const client = createGcloudClient(async (args) => {
|
||||
commands.push(args)
|
||||
return token
|
||||
})
|
||||
await expect(client.identityToken?.('https://relay.example/admin')).resolves.toBe(token)
|
||||
await expect(client.identityToken?.('https://relay.example/admin')).resolves.toBe(token)
|
||||
expect(commands).toEqual([
|
||||
[
|
||||
'auth',
|
||||
'print-identity-token',
|
||||
'--audiences=https://relay.example/admin',
|
||||
'--include-email'
|
||||
]
|
||||
])
|
||||
})
|
||||
})
|
||||
@@ -1,77 +0,0 @@
|
||||
import { execFile } from 'node:child_process'
|
||||
import { promisify } from 'node:util'
|
||||
|
||||
const execFileAsync = promisify(execFile)
|
||||
const TOKEN_PATTERN = /^[A-Za-z0-9._~+\/-]{32,8192}$/
|
||||
|
||||
export type GcloudClient = {
|
||||
accessToken(): Promise<string>
|
||||
identityToken?(audience: string): Promise<string>
|
||||
}
|
||||
|
||||
export class GcloudCommandError extends Error {
|
||||
constructor(readonly operation: string) {
|
||||
super(`${operation} is unavailable`)
|
||||
}
|
||||
}
|
||||
|
||||
async function runGcloud(args: string[]): Promise<string> {
|
||||
try {
|
||||
const result = await execFileAsync('gcloud', args, {
|
||||
encoding: 'utf8',
|
||||
timeout: 90_000,
|
||||
maxBuffer: 8 * 1024 * 1024,
|
||||
env: {
|
||||
...process.env,
|
||||
CLOUDSDK_COMPONENT_MANAGER_DISABLE_UPDATE_CHECK: '1',
|
||||
CLOUDSDK_CORE_DISABLE_PROMPTS: '1',
|
||||
CLOUDSDK_CORE_DISABLE_USAGE_REPORTING: '1'
|
||||
}
|
||||
})
|
||||
return result.stdout.trim()
|
||||
} catch {
|
||||
// Gcloud stderr can echo command context; keep dashboard errors intentionally non-sensitive.
|
||||
throw new GcloudCommandError(`gcloud ${args.slice(0, 3).join(' ')}`)
|
||||
}
|
||||
}
|
||||
|
||||
export function createGcloudClient(
|
||||
tokenCommand: (args: string[]) => Promise<string> = runGcloud
|
||||
): GcloudClient {
|
||||
let cachedToken: { value: string; expiresAt: number } | null = null
|
||||
const identityTokens = new Map<string, { value: string; expiresAt: number }>()
|
||||
let pendingToken: Promise<string> | null = null
|
||||
return {
|
||||
async accessToken(): Promise<string> {
|
||||
if (cachedToken && cachedToken.expiresAt > Date.now()) return cachedToken.value
|
||||
if (pendingToken) return await pendingToken
|
||||
// One refresh avoids concurrent gcloud processes contending on the local credential store.
|
||||
pendingToken = tokenCommand(['auth', 'print-access-token'])
|
||||
.then((token) => {
|
||||
if (!TOKEN_PATTERN.test(token)) throw new GcloudCommandError('gcloud access token')
|
||||
cachedToken = { value: token, expiresAt: Date.now() + 5 * 60_000 }
|
||||
return token
|
||||
})
|
||||
.finally(() => { pendingToken = null })
|
||||
return await pendingToken
|
||||
},
|
||||
async identityToken(audience: string): Promise<string> {
|
||||
const cached = identityTokens.get(audience)
|
||||
if (cached && cached.expiresAt > Date.now()) return cached.value
|
||||
const token = await tokenCommand([
|
||||
'auth',
|
||||
'print-identity-token',
|
||||
`--audiences=${audience}`,
|
||||
'--include-email'
|
||||
])
|
||||
if (
|
||||
token.length > 8_192 ||
|
||||
!/^[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+$/.test(token)
|
||||
) {
|
||||
throw new GcloudCommandError('gcloud identity token')
|
||||
}
|
||||
identityTokens.set(audience, { value: token, expiresAt: Date.now() + 5 * 60_000 })
|
||||
return token
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,77 +0,0 @@
|
||||
import { execFile } from 'node:child_process'
|
||||
import { promisify } from 'node:util'
|
||||
import { z } from 'zod'
|
||||
import { relayRepositoryApiPath } from './relay-repository.js'
|
||||
|
||||
const execFileAsync = promisify(execFile)
|
||||
|
||||
const WorkflowRunSchema = z.object({
|
||||
id: z.number().int().positive(),
|
||||
name: z.string(),
|
||||
event: z.string(),
|
||||
status: z.string(),
|
||||
conclusion: z.string().nullable(),
|
||||
head_sha: z.string(),
|
||||
html_url: z.string().url(),
|
||||
created_at: z.string(),
|
||||
updated_at: z.string()
|
||||
})
|
||||
|
||||
const WorkflowRunsSchema = z.object({
|
||||
workflow_runs: z.array(WorkflowRunSchema)
|
||||
})
|
||||
|
||||
export type RelayWorkflowRun = {
|
||||
id: number
|
||||
name: string
|
||||
status: string
|
||||
conclusion: string | null
|
||||
headSha: string
|
||||
url: string
|
||||
createdAt: string
|
||||
updatedAt: string
|
||||
}
|
||||
|
||||
const RELAY_WORKFLOW_PATTERN = /(Relay|Auth|Power)/i
|
||||
|
||||
export async function readRelayWorkflowRuns(): Promise<RelayWorkflowRun[]> {
|
||||
let stdout: string
|
||||
try {
|
||||
const result = await execFileAsync(
|
||||
'gh',
|
||||
[
|
||||
'api',
|
||||
'--method',
|
||||
'GET',
|
||||
relayRepositoryApiPath('actions/runs'),
|
||||
'-f',
|
||||
'per_page=100'
|
||||
],
|
||||
{ encoding: 'utf8', timeout: 30_000, maxBuffer: 4 * 1024 * 1024 }
|
||||
)
|
||||
stdout = result.stdout
|
||||
} catch {
|
||||
throw new Error('GitHub workflow history is unavailable')
|
||||
}
|
||||
const runs = WorkflowRunsSchema.parse(JSON.parse(stdout) as unknown).workflow_runs
|
||||
const counts = new Map<string, number>()
|
||||
return runs
|
||||
.filter((run) => {
|
||||
if (!RELAY_WORKFLOW_PATTERN.test(run.name)) return false
|
||||
const count = counts.get(run.name) ?? 0
|
||||
if (count >= 2) return false
|
||||
counts.set(run.name, count + 1)
|
||||
return true
|
||||
})
|
||||
.slice(0, 12)
|
||||
.map((run) => ({
|
||||
id: run.id,
|
||||
name: run.name,
|
||||
status: run.status,
|
||||
conclusion: run.conclusion,
|
||||
headSha: run.head_sha.slice(0, 8),
|
||||
url: run.html_url,
|
||||
createdAt: run.created_at,
|
||||
updatedAt: run.updated_at
|
||||
}))
|
||||
}
|
||||
@@ -1,874 +0,0 @@
|
||||
import { mkdtempSync, rmSync, writeFileSync } from 'node:fs'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { join } from 'node:path'
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest'
|
||||
import {
|
||||
livePreflightGcloud,
|
||||
runIncidentLivePreflight
|
||||
} from './incident-live-preflight-cli.js'
|
||||
import {
|
||||
INCIDENT_MONITOR_THRESHOLDS,
|
||||
type IncidentSample
|
||||
} from './incident-monitor.js'
|
||||
import { relayOpsEnvironment } from './environment-config.js'
|
||||
import type { AdmissionSelector } from './incident-selector.js'
|
||||
|
||||
const directories: string[] = []
|
||||
const now = Date.parse('2026-07-28T12:00:00.000Z')
|
||||
const selector = {
|
||||
generation: 1,
|
||||
membership: {
|
||||
existingOnly: ['production-gce-c1'],
|
||||
migrationOnly: [],
|
||||
general: []
|
||||
}
|
||||
}
|
||||
|
||||
function stateFile(
|
||||
migrationPolicy: 'strict' | 'recover-forward' | 'capacity-transition' = 'strict',
|
||||
overrides: Record<string, unknown> = {}
|
||||
): string {
|
||||
const directory = mkdtempSync(join(tmpdir(), 'relay-live-preflight-'))
|
||||
directories.push(directory)
|
||||
const path = join(directory, 'state.json')
|
||||
const expectedSelector = migrationPolicy === 'capacity-transition'
|
||||
? {
|
||||
generation: 1,
|
||||
membership: {
|
||||
existingOnly: [],
|
||||
migrationOnly: [],
|
||||
general: ['production-gce-c1']
|
||||
}
|
||||
}
|
||||
: selector
|
||||
writeFileSync(path, JSON.stringify({
|
||||
schemaVersion: 4,
|
||||
environment: 'production',
|
||||
expectedSelector,
|
||||
migrationPolicy,
|
||||
recoverySourceCellId:
|
||||
migrationPolicy === 'recover-forward' ? 'production-gce-c1' : null,
|
||||
capacityCellId:
|
||||
migrationPolicy === 'capacity-transition' ? 'production-gce-c1' : null,
|
||||
preDrainDryRun: true,
|
||||
startedAt: new Date(now - 17 * 60_000).toISOString(),
|
||||
windowStartedAt: new Date(now - 16 * 60_000).toISOString(),
|
||||
durationMinutes: 15,
|
||||
intervalMs: 60_000,
|
||||
sampleCount: 16,
|
||||
lastSampleAt: new Date(now - 60_007).toISOString(),
|
||||
frozenAt: null,
|
||||
completedAt: new Date(now - 60_000).toISOString(),
|
||||
...overrides
|
||||
}))
|
||||
return path
|
||||
}
|
||||
|
||||
// Every configured production cell, in the lexicographic order
|
||||
// normalizeSelectorMembership canonicalises to. Derived from the same durable
|
||||
// Terraform config the override path reads, so a new cell cannot strand these.
|
||||
const configuredCellIds = relayOpsEnvironment('production').cells.map(
|
||||
(cell) => cell.cellId
|
||||
)
|
||||
const canonicalCellIds = [...configuredCellIds].sort()
|
||||
const canonicalMembership = {
|
||||
existingOnly: canonicalCellIds,
|
||||
migrationOnly: [],
|
||||
general: []
|
||||
}
|
||||
|
||||
// What the director reports: a normalised selector, never an echo of what the
|
||||
// caller expected. An order-sensitive comparison only holds if the override path
|
||||
// canonicalises its own input the same way.
|
||||
function canonicalSample(generation = 1): IncidentSample {
|
||||
const next = sample()
|
||||
next.selector = { generation, membership: canonicalMembership }
|
||||
return next
|
||||
}
|
||||
|
||||
function membershipFile(
|
||||
membership: Record<string, string[]> = canonicalMembership
|
||||
): string {
|
||||
const directory = mkdtempSync(join(tmpdir(), 'relay-live-preflight-selector-'))
|
||||
directories.push(directory)
|
||||
const path = join(directory, 'selector.json')
|
||||
writeFileSync(path, JSON.stringify(membership))
|
||||
return path
|
||||
}
|
||||
|
||||
function sample(): IncidentSample {
|
||||
const observedAt = new Date(now).toISOString()
|
||||
const signal = (value: number) => ({ value, observedAt })
|
||||
return {
|
||||
collectedAt: observedAt,
|
||||
selector,
|
||||
expectedSelector: selector,
|
||||
cells: [{
|
||||
cellId: 'production-gce-c1',
|
||||
region: 'us-central1',
|
||||
runtimeKnown: true,
|
||||
powered: true,
|
||||
expectedAdmissionState: 'existing-only'
|
||||
}],
|
||||
sources: {
|
||||
'active-probe': {
|
||||
observedAt,
|
||||
signals: {
|
||||
'director.health': signal(1),
|
||||
'director.ready': signal(1),
|
||||
'director.latency_ms': signal(1),
|
||||
'auth.health': signal(1),
|
||||
'auth.ready': signal(1),
|
||||
'auth.latency_ms': signal(1),
|
||||
'cell.production-gce-c1.health': signal(1),
|
||||
'cell.production-gce-c1.ready': signal(1),
|
||||
'cell.production-gce-c1.latency_ms': signal(1)
|
||||
}
|
||||
},
|
||||
'cloud-monitoring': {
|
||||
observedAt,
|
||||
signals: {
|
||||
'cloud_sql.cpu': signal(0.1),
|
||||
'cloud_sql.memory': signal(0.1),
|
||||
'cloud_sql.backends': signal(1),
|
||||
'cloud_sql.lock_waits': signal(0),
|
||||
'cloud_sql.deadlocks': signal(0),
|
||||
'director.instances': signal(5),
|
||||
'director.cpu': signal(0.1),
|
||||
'director.memory': signal(0.1),
|
||||
'director.concurrency': signal(1),
|
||||
'director.errors': signal(0),
|
||||
'auth.errors': signal(0)
|
||||
}
|
||||
},
|
||||
'relay-logs': {
|
||||
observedAt,
|
||||
signals: {
|
||||
'relay.pool_waiting': signal(0),
|
||||
'relay.pool_wait_ms': signal(0),
|
||||
'relay.postgres_retries': signal(0),
|
||||
'relay.postgres_retry_exhausted': signal(0),
|
||||
'cell.production-gce-c1.connections': signal(1),
|
||||
'cell.production-gce-c1.queued_bytes': signal(0)
|
||||
}
|
||||
},
|
||||
'director-admin': {
|
||||
observedAt,
|
||||
signals: {
|
||||
'cell.production-gce-c1.admission_state': signal(0),
|
||||
'cell.production-gce-c1.heartbeat_fresh': signal(1),
|
||||
'cell.production-gce-c1.heartbeat_age_ms': signal(1),
|
||||
'cell.production-gce-c1.migration_blocked': signal(0),
|
||||
'cell.production-gce-c1.migration_target_inactive': signal(0)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
afterEach(() => {
|
||||
for (const directory of directories.splice(0)) {
|
||||
rmSync(directory, { recursive: true, force: true })
|
||||
}
|
||||
})
|
||||
|
||||
describe('relay incident live preflight', () => {
|
||||
it('accepts the package-manager argument separator', async () => {
|
||||
await expect(runIncidentLivePreflight(
|
||||
['--', '--state-file', stateFile()],
|
||||
{ now: () => now, collect: async () => sample() }
|
||||
)).resolves.toBeUndefined()
|
||||
})
|
||||
|
||||
it('accepts one complete fresh green sample', async () => {
|
||||
await expect(runIncidentLivePreflight(
|
||||
['--state-file', stateFile()],
|
||||
{ now: () => now, collect: async () => sample() }
|
||||
)).resolves.toBeUndefined()
|
||||
})
|
||||
|
||||
it('rejects monitor evidence beyond the 35-minute lineage bound', async () => {
|
||||
const path = stateFile('strict', {
|
||||
startedAt: new Date(now - 36 * 60_000 - 1).toISOString()
|
||||
})
|
||||
await expect(runIncidentLivePreflight(
|
||||
['--state-file', path],
|
||||
{ now: () => now, collect: async () => sample() }
|
||||
)).rejects.toThrow('monitor evidence is incomplete or stale')
|
||||
})
|
||||
|
||||
it('scales the evidence age bound by same-cap wave index', async () => {
|
||||
const agedState = (ageMs: number) => stateFile('strict', {
|
||||
startedAt: new Date(now - ageMs - 17 * 60_000).toISOString(),
|
||||
windowStartedAt: new Date(now - ageMs - 16 * 60_000).toISOString(),
|
||||
lastSampleAt: new Date(now - ageMs - 7).toISOString(),
|
||||
completedAt: new Date(now - ageMs).toISOString()
|
||||
})
|
||||
const deps = { now: () => now, collect: async () => sample() }
|
||||
// One predecessor cell roll (~16 min) exceeds wave 0 but fits wave 1.
|
||||
const oneRollOld = agedState(17 * 60_000)
|
||||
await expect(runIncidentLivePreflight(
|
||||
['--state-file', oneRollOld], deps
|
||||
)).rejects.toThrow('monitor evidence is incomplete or stale')
|
||||
await expect(runIncidentLivePreflight(
|
||||
['--state-file', oneRollOld, '--wave-index', '0'], deps
|
||||
)).rejects.toThrow('monitor evidence is incomplete or stale')
|
||||
await expect(runIncidentLivePreflight(
|
||||
['--state-file', oneRollOld, '--wave-index', '1'], deps
|
||||
)).resolves.toBeUndefined()
|
||||
// Wave 0 edges: the 10-minute bound covers same-cap job start-up latency.
|
||||
await expect(runIncidentLivePreflight(
|
||||
['--state-file', agedState(10 * 60_000), '--wave-index', '0'], deps
|
||||
)).resolves.toBeUndefined()
|
||||
await expect(runIncidentLivePreflight(
|
||||
['--state-file', agedState(10 * 60_000 + 1), '--wave-index', '0'], deps
|
||||
)).rejects.toThrow('monitor evidence is incomplete or stale')
|
||||
// Both edges of one predecessor job timeout: 10min + 75min exactly.
|
||||
await expect(runIncidentLivePreflight(
|
||||
['--state-file', agedState(85 * 60_000), '--wave-index', '1'], deps
|
||||
)).resolves.toBeUndefined()
|
||||
await expect(runIncidentLivePreflight(
|
||||
['--state-file', agedState(85 * 60_000 + 1), '--wave-index', '1'], deps
|
||||
)).rejects.toThrow('monitor evidence is incomplete or stale')
|
||||
await expect(runIncidentLivePreflight(
|
||||
['--state-file', agedState(160 * 60_000), '--wave-index', '2'], deps
|
||||
)).resolves.toBeUndefined()
|
||||
await expect(runIncidentLivePreflight(
|
||||
['--state-file', agedState(160 * 60_000 + 1), '--wave-index', '2'], deps
|
||||
)).rejects.toThrow('monitor evidence is incomplete or stale')
|
||||
await expect(runIncidentLivePreflight(
|
||||
['--state-file', agedState(235 * 60_000), '--wave-index', '3'], deps
|
||||
)).resolves.toBeUndefined()
|
||||
await expect(runIncidentLivePreflight(
|
||||
['--state-file', agedState(235 * 60_000 + 1), '--wave-index', '3'], deps
|
||||
)).rejects.toThrow('monitor evidence is incomplete or stale')
|
||||
// The last cell of a ten-cell same-cap batch: 10min + 9 * 75min exactly.
|
||||
await expect(runIncidentLivePreflight(
|
||||
['--state-file', agedState(685 * 60_000), '--wave-index', '9'], deps
|
||||
)).resolves.toBeUndefined()
|
||||
await expect(runIncidentLivePreflight(
|
||||
['--state-file', agedState(685 * 60_000 + 1), '--wave-index', '9'], deps
|
||||
)).rejects.toThrow('monitor evidence is incomplete or stale')
|
||||
// The wave index is a strict single-use 0-9 argument.
|
||||
await expect(runIncidentLivePreflight(
|
||||
['--state-file', stateFile(), '--wave-index', '10'], deps
|
||||
)).rejects.toThrow('usage:')
|
||||
await expect(runIncidentLivePreflight(
|
||||
['--state-file', stateFile(), '--wave-index', ''], deps
|
||||
)).rejects.toThrow('usage:')
|
||||
await expect(runIncidentLivePreflight(
|
||||
['--state-file', stateFile(), '--wave-index', '1', '--wave-index', '1'],
|
||||
deps
|
||||
)).rejects.toThrow('usage:')
|
||||
})
|
||||
|
||||
it('expects the wave-adjusted live selector generation', async () => {
|
||||
const agedPath = stateFile('strict', {
|
||||
startedAt: new Date(now - 34 * 60_000).toISOString(),
|
||||
windowStartedAt: new Date(now - 33 * 60_000).toISOString(),
|
||||
lastSampleAt: new Date(now - 17 * 60_000 - 7).toISOString(),
|
||||
completedAt: new Date(now - 17 * 60_000).toISOString()
|
||||
})
|
||||
const liveAt = (generation: number) =>
|
||||
async (expectedSelector: AdmissionSelector) => ({
|
||||
...sample(),
|
||||
selector: { ...selector, generation },
|
||||
expectedSelector
|
||||
})
|
||||
// One predecessor roll advanced the live selector by exactly 2.
|
||||
await expect(runIncidentLivePreflight(
|
||||
['--state-file', agedPath, '--wave-index', '1'],
|
||||
{ now: () => now, collect: liveAt(selector.generation + 2) }
|
||||
)).resolves.toBeUndefined()
|
||||
// The sealed pre-roll generation must no longer satisfy wave 1.
|
||||
await expect(runIncidentLivePreflight(
|
||||
['--state-file', agedPath, '--wave-index', '1'],
|
||||
{ now: () => now, collect: liveAt(selector.generation) }
|
||||
)).rejects.toThrow('director-admin/selector_mismatch')
|
||||
// Wave 0 still expects the sealed generation itself.
|
||||
await expect(runIncidentLivePreflight(
|
||||
['--state-file', stateFile()],
|
||||
{ now: () => now, collect: liveAt(selector.generation) }
|
||||
)).resolves.toBeUndefined()
|
||||
})
|
||||
|
||||
it('fails closed on a live threshold breach', async () => {
|
||||
const unhealthy = sample()
|
||||
unhealthy.sources['cloud-monitoring']!.signals['cloud_sql.cpu']!.value = 0.9
|
||||
await expect(runIncidentLivePreflight(
|
||||
['--state-file', stateFile()],
|
||||
{ now: () => now, collect: async () => unhealthy }
|
||||
)).rejects.toThrow('cloud-monitoring/threshold_max')
|
||||
})
|
||||
|
||||
// Why: a frozen wave has to name what froze it without re-reading the sample.
|
||||
it('names the signal and its numbers in the failure message', async () => {
|
||||
const slowCell = sample()
|
||||
slowCell.sources['active-probe']!.signals[
|
||||
'cell.production-gce-c1.latency_ms'
|
||||
]!.value = 2_568
|
||||
// A cell probe now re-samples before it fails a wave, so the wait is injected;
|
||||
// this cell stays slow on every sample and still names what stopped it.
|
||||
await expect(runIncidentLivePreflight(
|
||||
['--state-file', stateFile()],
|
||||
{ now: () => now, collect: async () => slowCell, wait: async () => {} }
|
||||
)).rejects.toThrow(
|
||||
'relay live preflight failed: active-probe/threshold_max cell.production-gce-c1.latency_ms observed=2568 threshold=2000'
|
||||
)
|
||||
|
||||
// A failure with no signal keeps the source/code token and drops the rest.
|
||||
const stale = sample()
|
||||
stale.sources['active-probe']!.observedAt = new Date(now - 60_001).toISOString()
|
||||
await expect(runIncidentLivePreflight(
|
||||
['--state-file', stateFile()],
|
||||
{ now: () => now, collect: async () => stale }
|
||||
)).rejects.toThrow(
|
||||
'relay live preflight failed: active-probe/source_stale observed=60001 threshold=60000'
|
||||
)
|
||||
})
|
||||
|
||||
// Why: this one sample decides a mutating wave, so an Asia cell's ~30 s
|
||||
// "no healthy upstream" window could still fail a wave here even after the
|
||||
// 15-minute gate learned to ride it out.
|
||||
describe('cell probe tolerance', () => {
|
||||
const tolerance = INCIDENT_MONITOR_THRESHOLDS.cellProbeToleranceSamples
|
||||
|
||||
// Serves `badSamples` unhealthy cell readings, then healthy ones.
|
||||
const downThen = (badSamples: number) => {
|
||||
let index = 0
|
||||
return async () => {
|
||||
const next = sample()
|
||||
if (index++ < badSamples) {
|
||||
next.sources['active-probe']!.signals['cell.production-gce-c1.health']!
|
||||
.value = 0
|
||||
next.sources['active-probe']!.signals['cell.production-gce-c1.ready']!
|
||||
.value = 0
|
||||
}
|
||||
return next
|
||||
}
|
||||
}
|
||||
|
||||
it('re-samples through a probe outage within the tolerance', async () => {
|
||||
const waits: number[] = []
|
||||
await expect(runIncidentLivePreflight(
|
||||
['--state-file', stateFile()],
|
||||
{
|
||||
now: () => now,
|
||||
collect: downThen(tolerance),
|
||||
wait: async (ms) => {
|
||||
waits.push(ms)
|
||||
}
|
||||
}
|
||||
)).resolves.toBeUndefined()
|
||||
expect(waits).toHaveLength(tolerance)
|
||||
})
|
||||
|
||||
it('fails the wave once the probe outage outlasts the tolerance', async () => {
|
||||
await expect(runIncidentLivePreflight(
|
||||
['--state-file', stateFile()],
|
||||
{
|
||||
now: () => now,
|
||||
collect: downThen(tolerance + 1),
|
||||
wait: async () => {}
|
||||
}
|
||||
)).rejects.toThrow('active-probe/threshold_equal cell.production-gce-c1.health')
|
||||
})
|
||||
|
||||
it('does not re-sample a director probe failure', async () => {
|
||||
let samples = 0
|
||||
const down = async () => {
|
||||
samples++
|
||||
const next = sample()
|
||||
next.sources['active-probe']!.signals['director.health']!.value = 0
|
||||
return next
|
||||
}
|
||||
await expect(runIncidentLivePreflight(
|
||||
['--state-file', stateFile()],
|
||||
{ now: () => now, collect: down, wait: async () => {} }
|
||||
)).rejects.toThrow('active-probe/threshold_equal director.health')
|
||||
expect(samples).toBe(1)
|
||||
})
|
||||
|
||||
it('does not re-sample a non-probe threshold failure', async () => {
|
||||
let samples = 0
|
||||
const hot = async () => {
|
||||
samples++
|
||||
const next = sample()
|
||||
next.sources['cloud-monitoring']!.signals['cloud_sql.cpu']!.value = 0.9
|
||||
return next
|
||||
}
|
||||
await expect(runIncidentLivePreflight(
|
||||
['--state-file', stateFile()],
|
||||
{ now: () => now, collect: hot, wait: async () => {} }
|
||||
)).rejects.toThrow('cloud-monitoring/threshold_max')
|
||||
expect(samples).toBe(1)
|
||||
})
|
||||
})
|
||||
|
||||
// Why: on 2026-09-17 a canary wave died because one director admin read
|
||||
// returned 404 for a 2 s Cloud SQL pool timeout. Collecting the sample is not
|
||||
// a health verdict, so a thrown collector spends an attempt instead.
|
||||
describe('collector failures', () => {
|
||||
it('re-samples after a thrown collector and then passes', async () => {
|
||||
const waits: number[] = []
|
||||
let attempts = 0
|
||||
await expect(runIncidentLivePreflight(
|
||||
['--state-file', stateFile()],
|
||||
{
|
||||
now: () => now,
|
||||
collect: async () => {
|
||||
attempts++
|
||||
if (attempts === 1) throw new Error('Relay admin telemetry returned 404')
|
||||
return sample()
|
||||
},
|
||||
wait: async (ms) => {
|
||||
waits.push(ms)
|
||||
}
|
||||
}
|
||||
)).resolves.toBeUndefined()
|
||||
expect(attempts).toBe(2)
|
||||
expect(waits).toEqual([15_000])
|
||||
})
|
||||
|
||||
it('fails the wave when every attempt throws, naming the collector', async () => {
|
||||
let attempts = 0
|
||||
await expect(runIncidentLivePreflight(
|
||||
['--state-file', stateFile()],
|
||||
{
|
||||
now: () => now,
|
||||
collect: async () => {
|
||||
attempts++
|
||||
throw new Error('Relay admin telemetry returned 404')
|
||||
},
|
||||
wait: async () => {}
|
||||
}
|
||||
)).rejects.toThrow(
|
||||
'relay live preflight failed: collector: Relay admin telemetry returned 404'
|
||||
)
|
||||
expect(attempts).toBe(1 + INCIDENT_MONITOR_THRESHOLDS.cellProbeToleranceSamples)
|
||||
})
|
||||
|
||||
it('does not re-sample a collector failure past the evidence-age budget', async () => {
|
||||
const agedPath = stateFile('strict', {
|
||||
startedAt: new Date(now - 26 * 60_000).toISOString(),
|
||||
windowStartedAt: new Date(now - 25 * 60_000).toISOString(),
|
||||
lastSampleAt: new Date(now - 10 * 60_000 + 7).toISOString(),
|
||||
completedAt: new Date(now - 10 * 60_000 + 14).toISOString()
|
||||
})
|
||||
let attempts = 0
|
||||
await expect(runIncidentLivePreflight(
|
||||
['--state-file', agedPath],
|
||||
{
|
||||
now: () => now,
|
||||
collect: async () => {
|
||||
attempts++
|
||||
throw new Error('Relay admin telemetry returned 404')
|
||||
},
|
||||
wait: async () => {}
|
||||
}
|
||||
)).rejects.toThrow('relay live preflight failed: collector:')
|
||||
expect(attempts).toBe(1)
|
||||
})
|
||||
})
|
||||
|
||||
it('enforces the signed migration policy', async () => {
|
||||
const inactiveTarget = sample()
|
||||
inactiveTarget.sources['director-admin']!.signals[
|
||||
'cell.production-gce-c1.migration_target_inactive'
|
||||
]!.value = 30
|
||||
await expect(runIncidentLivePreflight(
|
||||
['--state-file', stateFile()],
|
||||
{ now: () => now, collect: async () => inactiveTarget }
|
||||
)).rejects.toThrow('director-admin/threshold_max')
|
||||
await expect(runIncidentLivePreflight(
|
||||
['--state-file', stateFile('recover-forward')],
|
||||
{ now: () => now, collect: async () => inactiveTarget }
|
||||
)).resolves.toBeUndefined()
|
||||
inactiveTarget.sources['director-admin']!.signals[
|
||||
'cell.production-gce-c1.migration_blocked'
|
||||
]!.value = 1
|
||||
await expect(runIncidentLivePreflight(
|
||||
['--state-file', stateFile('recover-forward')],
|
||||
{ now: () => now, collect: async () => inactiveTarget }
|
||||
)).rejects.toThrow('director-admin/threshold_max')
|
||||
})
|
||||
|
||||
it('binds capacity-transition evidence to its general cell', async () => {
|
||||
const capacitySample = sample()
|
||||
const capacitySelector = {
|
||||
generation: 1,
|
||||
membership: {
|
||||
existingOnly: [],
|
||||
migrationOnly: [],
|
||||
general: ['production-gce-c1']
|
||||
}
|
||||
}
|
||||
capacitySample.selector = capacitySelector
|
||||
capacitySample.expectedSelector = capacitySelector
|
||||
capacitySample.cells[0]!.expectedAdmissionState = 'general'
|
||||
capacitySample.sources['director-admin']!.signals[
|
||||
'cell.production-gce-c1.admission_state'
|
||||
]!.value = 2
|
||||
await expect(runIncidentLivePreflight(
|
||||
['--state-file', stateFile('capacity-transition')],
|
||||
{ now: () => now, collect: async () => capacitySample }
|
||||
)).resolves.toBeUndefined()
|
||||
capacitySample.sources['director-admin']!.signals[
|
||||
'cell.production-gce-c1.migration_target_inactive'
|
||||
]!.value = 1
|
||||
await expect(runIncidentLivePreflight(
|
||||
['--state-file', stateFile('capacity-transition')],
|
||||
{ now: () => now, collect: async () => capacitySample }
|
||||
)).rejects.toThrow('director-admin/threshold_max')
|
||||
})
|
||||
|
||||
it('rejects stale live evidence', async () => {
|
||||
const stale = sample()
|
||||
stale.sources['active-probe']!.observedAt = new Date(now - 60_001).toISOString()
|
||||
await expect(runIncidentLivePreflight(
|
||||
['--state-file', stateFile()],
|
||||
{ now: () => now, collect: async () => stale }
|
||||
)).rejects.toThrow('active-probe/source_stale')
|
||||
})
|
||||
|
||||
it('retries freshness-only failures when explicitly requested', async () => {
|
||||
const stale = sample()
|
||||
stale.sources['cloud-monitoring']!.signals['cloud_sql.cpu']!.observedAt =
|
||||
new Date(now - (INCIDENT_MONITOR_THRESHOLDS.cloudDataMaxAgeMs + 1)).toISOString()
|
||||
const missing = sample()
|
||||
delete missing.sources['relay-logs']
|
||||
const collect = vi.fn()
|
||||
.mockResolvedValueOnce(stale)
|
||||
.mockResolvedValueOnce(missing)
|
||||
.mockResolvedValueOnce(sample())
|
||||
const wait = vi.fn(async () => undefined)
|
||||
await expect(runIncidentLivePreflight(
|
||||
['--state-file', stateFile(), '--retry-freshness'],
|
||||
{ now: () => now, collect, wait }
|
||||
)).resolves.toBeUndefined()
|
||||
expect(collect).toHaveBeenCalledTimes(3)
|
||||
expect(wait).toHaveBeenCalledTimes(2)
|
||||
expect(wait).toHaveBeenNthCalledWith(1, 15_000)
|
||||
expect(wait).toHaveBeenNthCalledWith(2, 15_000)
|
||||
})
|
||||
|
||||
it('retries a first-wave stale sample and passes on the fresh one', async () => {
|
||||
const stale = sample()
|
||||
stale.sources['cloud-monitoring']!.signals['cloud_sql.cpu']!.observedAt =
|
||||
new Date(now - (INCIDENT_MONITOR_THRESHOLDS.cloudDataMaxAgeMs + 1)).toISOString()
|
||||
const collect = vi.fn().mockResolvedValueOnce(stale).mockResolvedValueOnce(sample())
|
||||
const wait = vi.fn(async () => undefined)
|
||||
await expect(runIncidentLivePreflight(
|
||||
['--state-file', stateFile(), '--wave-index', '0', '--retry-freshness'],
|
||||
{ now: () => now, collect, wait }
|
||||
)).resolves.toBeUndefined()
|
||||
expect(collect).toHaveBeenCalledTimes(2)
|
||||
expect(wait).toHaveBeenCalledOnce()
|
||||
})
|
||||
|
||||
it('stops retrying when the next wait would exceed the evidence-age bound', async () => {
|
||||
const completedAt = now - 590_000
|
||||
const stale = sample()
|
||||
stale.sources['cloud-monitoring']!.observedAt = new Date(now - (INCIDENT_MONITOR_THRESHOLDS.cloudDataMaxAgeMs + 1)).toISOString()
|
||||
const collect = vi.fn(async () => stale)
|
||||
const wait = vi.fn(async () => undefined)
|
||||
await expect(runIncidentLivePreflight(
|
||||
['--state-file', stateFile('strict', {
|
||||
startedAt: new Date(completedAt - 17 * 60_000).toISOString(),
|
||||
windowStartedAt: new Date(completedAt - 16 * 60_000).toISOString(),
|
||||
lastSampleAt: new Date(completedAt - 30_000).toISOString(),
|
||||
completedAt: new Date(completedAt).toISOString()
|
||||
}), '--retry-freshness'],
|
||||
{ now: () => now, collect, wait }
|
||||
)).rejects.toThrow('cloud-monitoring/source_stale')
|
||||
expect(collect).toHaveBeenCalledOnce()
|
||||
expect(wait).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('does not retry a threshold failure', async () => {
|
||||
const unhealthy = sample()
|
||||
unhealthy.sources['cloud-monitoring']!.signals['cloud_sql.cpu']!.value = 0.9
|
||||
unhealthy.sources['cloud-monitoring']!.signals['cloud_sql.cpu']!.observedAt =
|
||||
new Date(now - (INCIDENT_MONITOR_THRESHOLDS.cloudDataMaxAgeMs + 1)).toISOString()
|
||||
const collect = vi.fn(async () => unhealthy)
|
||||
const wait = vi.fn(async () => undefined)
|
||||
await expect(runIncidentLivePreflight(
|
||||
['--state-file', stateFile(), '--retry-freshness'],
|
||||
{ now: () => now, collect, wait }
|
||||
)).rejects.toThrow('cloud-monitoring/threshold_max')
|
||||
expect(collect).toHaveBeenCalledOnce()
|
||||
expect(wait).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('fails closed after the bounded freshness retry window', async () => {
|
||||
const stale = sample()
|
||||
stale.sources['cloud-monitoring']!.observedAt = new Date(now - (INCIDENT_MONITOR_THRESHOLDS.cloudDataMaxAgeMs + 1)).toISOString()
|
||||
const collect = vi.fn(async () => stale)
|
||||
const wait = vi.fn(async () => undefined)
|
||||
await expect(runIncidentLivePreflight(
|
||||
['--state-file', stateFile(), '--retry-freshness'],
|
||||
{ now: () => now, collect, wait }
|
||||
)).rejects.toThrow('cloud-monitoring/source_stale')
|
||||
expect(collect).toHaveBeenCalledTimes(5)
|
||||
expect(wait).toHaveBeenCalledTimes(4)
|
||||
})
|
||||
|
||||
|
||||
// Why: a same-cap wave has no sealed 15-minute evidence, so this live recheck
|
||||
// and the pre-drain sample after it are what stand between the dispatch and a
|
||||
// mutation. It must judge the fleet exactly as it does with evidence, and it
|
||||
// must never accept a half-specified selector.
|
||||
describe('dispatch selector without monitor state', () => {
|
||||
const overrideArgs = (extra: string[] = [], membership = canonicalMembership) => [
|
||||
'--no-monitor-state',
|
||||
'--expected-selector-generation', '1',
|
||||
'--selector-membership-file', membershipFile(membership),
|
||||
...extra
|
||||
]
|
||||
|
||||
// The director's reading, plus whatever the override path decided to expect.
|
||||
const liveCollect = (
|
||||
mutate: (next: IncidentSample) => IncidentSample = (next) => next
|
||||
) => async (expected: AdmissionSelector) => {
|
||||
const next = canonicalSample(expected.generation)
|
||||
next.expectedSelector = expected
|
||||
return mutate(next)
|
||||
}
|
||||
|
||||
it('accepts one complete fresh green sample with no sealed evidence', async () => {
|
||||
await expect(runIncidentLivePreflight(
|
||||
overrideArgs(),
|
||||
{ now: () => now, collect: liveCollect() }
|
||||
)).resolves.toBeUndefined()
|
||||
})
|
||||
|
||||
// Why: the live selector is normalised and the comparison is an ordered
|
||||
// stringify, so an operator's unsorted membership must canonicalise here or
|
||||
// every override wave reads as selector drift on a healthy fleet.
|
||||
it('canonicalises an unsorted operator membership', async () => {
|
||||
const shuffled = {
|
||||
existingOnly: [...canonicalCellIds].reverse(),
|
||||
migrationOnly: [],
|
||||
general: []
|
||||
}
|
||||
expect(shuffled.existingOnly).not.toEqual(canonicalCellIds)
|
||||
const seen: AdmissionSelector[] = []
|
||||
await expect(runIncidentLivePreflight(
|
||||
overrideArgs([], shuffled),
|
||||
{
|
||||
now: () => now,
|
||||
collect: async (expected) => {
|
||||
seen.push(expected)
|
||||
const next = canonicalSample(expected.generation)
|
||||
next.expectedSelector = expected
|
||||
return next
|
||||
}
|
||||
}
|
||||
)).resolves.toBeUndefined()
|
||||
expect(seen[0]!.membership.existingOnly).toEqual(canonicalCellIds)
|
||||
})
|
||||
|
||||
// Why: normalising is also what enforces every configured cell exactly once,
|
||||
// which a bare schema parse would have dropped.
|
||||
it('rejects a membership that is not every configured cell exactly once', async () => {
|
||||
const duplicated = {
|
||||
existingOnly: [...canonicalCellIds, canonicalCellIds[0] as string],
|
||||
migrationOnly: [],
|
||||
general: []
|
||||
}
|
||||
await expect(runIncidentLivePreflight(
|
||||
overrideArgs([], duplicated),
|
||||
{ now: () => now, collect: liveCollect() }
|
||||
)).rejects.toThrow('every configured cell exactly once')
|
||||
const missing = {
|
||||
existingOnly: canonicalCellIds.slice(1),
|
||||
migrationOnly: [],
|
||||
general: []
|
||||
}
|
||||
await expect(runIncidentLivePreflight(
|
||||
overrideArgs([], missing),
|
||||
{ now: () => now, collect: liveCollect() }
|
||||
)).rejects.toThrow('every configured cell exactly once')
|
||||
const unknown = {
|
||||
existingOnly: [...canonicalCellIds.slice(1), 'production-gce-c999'],
|
||||
migrationOnly: [],
|
||||
general: []
|
||||
}
|
||||
await expect(runIncidentLivePreflight(
|
||||
overrideArgs([], unknown),
|
||||
{ now: () => now, collect: liveCollect() }
|
||||
)).rejects.toThrow('every configured cell exactly once')
|
||||
})
|
||||
|
||||
it('fails closed on a live threshold breach', async () => {
|
||||
await expect(runIncidentLivePreflight(
|
||||
overrideArgs(),
|
||||
{
|
||||
now: () => now,
|
||||
collect: liveCollect((next) => {
|
||||
next.sources['cloud-monitoring']!.signals['cloud_sql.cpu']!.value = 0.99
|
||||
return next
|
||||
}),
|
||||
wait: async () => {}
|
||||
}
|
||||
)).rejects.toThrow('cloud-monitoring/threshold_max cloud_sql.cpu')
|
||||
})
|
||||
|
||||
it('fails closed on a live selector mismatch', async () => {
|
||||
await expect(runIncidentLivePreflight(
|
||||
overrideArgs(),
|
||||
{
|
||||
now: () => now,
|
||||
collect: liveCollect((next) => {
|
||||
next.selector = { ...next.selector, generation: 7 }
|
||||
return next
|
||||
}),
|
||||
wait: async () => {}
|
||||
}
|
||||
)).rejects.toThrow('selector_mismatch')
|
||||
})
|
||||
|
||||
it('expects the wave-adjusted live selector generation', async () => {
|
||||
const seen: AdmissionSelector[] = []
|
||||
await expect(runIncidentLivePreflight(
|
||||
overrideArgs(['--wave-index', '2']),
|
||||
{
|
||||
now: () => now,
|
||||
collect: async (expected) => {
|
||||
seen.push(expected)
|
||||
const next = canonicalSample(expected.generation)
|
||||
next.expectedSelector = expected
|
||||
return next
|
||||
}
|
||||
}
|
||||
)).resolves.toBeUndefined()
|
||||
expect(seen[0]!.generation).toBe(5)
|
||||
})
|
||||
|
||||
it('offsets by the wave delta the cell class declares', async () => {
|
||||
const generationFor = async (args: string[]) => {
|
||||
const seen: AdmissionSelector[] = []
|
||||
await expect(runIncidentLivePreflight(args, {
|
||||
now: () => now,
|
||||
collect: async (expected) => {
|
||||
seen.push(expected)
|
||||
const next = canonicalSample(expected.generation)
|
||||
next.expectedSelector = expected
|
||||
return next
|
||||
}
|
||||
})).resolves.toBeUndefined()
|
||||
return seen[0]!.generation
|
||||
}
|
||||
// A migration-only cell's wave isolates and restores nothing, so no predecessor moved it.
|
||||
expect(await generationFor(
|
||||
overrideArgs(['--wave-index', '2', '--selector-wave-delta', '0'])
|
||||
)).toBe(1)
|
||||
expect(await generationFor(
|
||||
overrideArgs(['--wave-index', '2', '--selector-wave-delta', '2'])
|
||||
)).toBe(5)
|
||||
})
|
||||
|
||||
it('rejects a selector wave delta no cell class produces', async () => {
|
||||
for (const delta of ['1', '3', '4', '', '-0', '02']) {
|
||||
await expect(runIncidentLivePreflight(
|
||||
overrideArgs(['--selector-wave-delta', delta]),
|
||||
{ now: () => now }
|
||||
)).rejects.toThrow('usage:')
|
||||
}
|
||||
await expect(runIncidentLivePreflight(
|
||||
overrideArgs(['--selector-wave-delta', '0', '--selector-wave-delta', '0']),
|
||||
{ now: () => now }
|
||||
)).rejects.toThrow('usage:')
|
||||
})
|
||||
|
||||
it('pins the strictest migration policy', async () => {
|
||||
// An inactive migration target is tolerable only under recover-forward,
|
||||
// and an override cannot elect that policy, so this must still fail.
|
||||
await expect(runIncidentLivePreflight(
|
||||
overrideArgs(),
|
||||
{
|
||||
now: () => now,
|
||||
collect: liveCollect((next) => {
|
||||
next.sources['director-admin']!.signals[
|
||||
'cell.production-gce-c1.migration_target_inactive'
|
||||
]!.value = 30
|
||||
return next
|
||||
}),
|
||||
wait: async () => {}
|
||||
}
|
||||
)).rejects.toThrow('director-admin/threshold_max')
|
||||
})
|
||||
|
||||
it('rejects a half-specified override', async () => {
|
||||
const cases: string[][] = [
|
||||
['--no-monitor-state'],
|
||||
['--no-monitor-state', '--expected-selector-generation', '1'],
|
||||
['--no-monitor-state', '--selector-membership-file', membershipFile()],
|
||||
// Mixing the two sources would let a caller pass sealed evidence it
|
||||
// never wants read.
|
||||
[
|
||||
'--no-monitor-state',
|
||||
'--expected-selector-generation', '1',
|
||||
'--selector-membership-file', membershipFile(),
|
||||
'--state-file', stateFile()
|
||||
],
|
||||
// Override arguments without the flag must not be silently ignored.
|
||||
['--state-file', stateFile(), '--expected-selector-generation', '1'],
|
||||
['--no-monitor-state', '--no-monitor-state'],
|
||||
['--expected-selector-generation', '1']
|
||||
]
|
||||
for (const args of cases) {
|
||||
await expect(runIncidentLivePreflight(
|
||||
args,
|
||||
{ now: () => now, collect: liveCollect() }
|
||||
)).rejects.toThrow('usage:')
|
||||
}
|
||||
})
|
||||
|
||||
it('rejects an unknown option and a negative generation', async () => {
|
||||
await expect(runIncidentLivePreflight(
|
||||
overrideArgs(['--skip-everything']),
|
||||
{ now: () => now, collect: liveCollect() }
|
||||
)).rejects.toThrow('usage:')
|
||||
await expect(runIncidentLivePreflight(
|
||||
[
|
||||
'--no-monitor-state',
|
||||
'--expected-selector-generation', '-1',
|
||||
'--selector-membership-file', membershipFile()
|
||||
],
|
||||
{ now: () => now, collect: liveCollect() }
|
||||
)).rejects.toThrow()
|
||||
})
|
||||
|
||||
it('re-samples a tolerable failure and then passes', async () => {
|
||||
let samples = 0
|
||||
await expect(runIncidentLivePreflight(
|
||||
overrideArgs(),
|
||||
{
|
||||
now: () => now,
|
||||
collect: liveCollect((next) => {
|
||||
samples++
|
||||
if (samples === 1) {
|
||||
next.sources['cloud-monitoring']!.signals['director.instances']!.value = 2
|
||||
}
|
||||
return next
|
||||
}),
|
||||
wait: async () => {}
|
||||
}
|
||||
)).resolves.toBeUndefined()
|
||||
expect(samples).toBe(2)
|
||||
})
|
||||
})
|
||||
|
||||
it('uses the supplied admin token without minting through gcloud', async () => {
|
||||
const identityToken = vi.fn(async () => 'minted.token.value')
|
||||
const gcloud = livePreflightGcloud(
|
||||
{ accessToken: async () => 'access-token', identityToken },
|
||||
{ ORCA_RELAY_ADMIN_ID_TOKEN: 'supplied.token.value' }
|
||||
)
|
||||
await expect(gcloud.identityToken!('audience')).resolves.toBe(
|
||||
'supplied.token.value'
|
||||
)
|
||||
expect(identityToken).not.toHaveBeenCalled()
|
||||
})
|
||||
})
|
||||
@@ -1,360 +0,0 @@
|
||||
import { readFile } from 'node:fs/promises'
|
||||
import { resolve } from 'node:path'
|
||||
import { pathToFileURL } from 'node:url'
|
||||
import { z } from 'zod'
|
||||
import { relayOpsEnvironment } from './environment-config.js'
|
||||
import { createGcloudClient } from './gcloud-client.js'
|
||||
import { suppliedIdentityToken } from './incident-monitor-cli.js'
|
||||
import {
|
||||
AdmissionSelectorSchema,
|
||||
normalizeSelectorMembership,
|
||||
SelectorMembershipSchema,
|
||||
type AdmissionSelector
|
||||
} from './incident-selector.js'
|
||||
import {
|
||||
describeIncidentFailure,
|
||||
evaluateIncidentSample,
|
||||
FRESHNESS_FAILURE_CODES,
|
||||
INCIDENT_MONITOR_THRESHOLDS,
|
||||
preDrainDryRunPassed,
|
||||
toleratedStreakKey,
|
||||
type IncidentSample
|
||||
} from './incident-monitor.js'
|
||||
import { createIncidentSampleCollector } from './incident-monitor-sources.js'
|
||||
|
||||
const FRESHNESS_RETRY_ATTEMPTS = 5
|
||||
const FRESHNESS_RETRY_INTERVAL_MS = 15_000
|
||||
// 10 min, not 5: a wave job reaches this check ~5 min after the monitor completes
|
||||
// (runner queue ~2 min, gate job ~80 s, checkout ~60 s); on 2026-09-17 a green
|
||||
// gate died at 302 s. The live samples below hold every wave to now.
|
||||
const MONITOR_EVIDENCE_MAX_AGE_MS = 10 * 60_000
|
||||
// Matches the wave job timeout-minutes; bounds each predecessor wave.
|
||||
const WAVE_PREDECESSOR_TIMEOUT_MS = 75 * 60_000
|
||||
// Widest any wave chain declares (same-cap's cell_1..cell_10); each job workflow
|
||||
// pins its own narrower range.
|
||||
const WAVE_INDEX_PATTERN = /^[0-9]$/
|
||||
// 2 for a general cell's isolate-and-restore wave, 0 for a migration-only cell's no-op pair.
|
||||
const SELECTOR_WAVE_DELTA_PATTERN = /^[02]$/
|
||||
|
||||
export function livePreflightGcloud(
|
||||
gcloud: ReturnType<typeof createGcloudClient>,
|
||||
environment: NodeJS.ProcessEnv = process.env
|
||||
): ReturnType<typeof createGcloudClient> {
|
||||
const token = suppliedIdentityToken(environment.ORCA_RELAY_ADMIN_ID_TOKEN)
|
||||
return token ? { ...gcloud, identityToken: async () => token } : gcloud
|
||||
}
|
||||
|
||||
const PreflightStateSchema = z.object({
|
||||
schemaVersion: z.literal(4),
|
||||
environment: z.literal('production'),
|
||||
expectedSelector: AdmissionSelectorSchema,
|
||||
migrationPolicy: z.enum(['strict', 'recover-forward', 'capacity-transition']),
|
||||
recoverySourceCellId: z.string().nullable(),
|
||||
capacityCellId: z.string().nullable(),
|
||||
preDrainDryRun: z.literal(true),
|
||||
startedAt: z.string(),
|
||||
windowStartedAt: z.string(),
|
||||
durationMinutes: z.literal(15),
|
||||
intervalMs: z.literal(60_000),
|
||||
sampleCount: z.number().int().min(16),
|
||||
lastSampleAt: z.string(),
|
||||
frozenAt: z.null(),
|
||||
completedAt: z.string()
|
||||
}).superRefine((state, context) => {
|
||||
const validRecovery =
|
||||
state.migrationPolicy === 'recover-forward' &&
|
||||
state.capacityCellId === null &&
|
||||
state.recoverySourceCellId !== null &&
|
||||
state.expectedSelector.membership.existingOnly.includes(
|
||||
state.recoverySourceCellId
|
||||
)
|
||||
const validStrict =
|
||||
state.migrationPolicy === 'strict' &&
|
||||
state.recoverySourceCellId === null &&
|
||||
state.capacityCellId === null
|
||||
const validCapacity =
|
||||
state.migrationPolicy === 'capacity-transition' &&
|
||||
state.recoverySourceCellId === null &&
|
||||
state.capacityCellId !== null &&
|
||||
state.expectedSelector.membership.general.includes(state.capacityCellId)
|
||||
if (!validRecovery && !validStrict && !validCapacity) {
|
||||
context.addIssue({
|
||||
code: 'custom',
|
||||
message: 'relay live preflight migration policy is invalid'
|
||||
})
|
||||
}
|
||||
})
|
||||
|
||||
const PREFLIGHT_USAGE =
|
||||
'usage: --state-file <verified-monitor-state> [--wave-index <0-3>]' +
|
||||
' [--selector-wave-delta <0|2>] [--retry-freshness]' +
|
||||
' | --no-monitor-state --expected-selector-generation <n>' +
|
||||
' --selector-membership-file <json> [--wave-index <0-3>]' +
|
||||
' [--selector-wave-delta <0|2>]'
|
||||
|
||||
const VALUE_OPTIONS = new Set([
|
||||
'--state-file',
|
||||
'--wave-index',
|
||||
'--selector-wave-delta',
|
||||
'--expected-selector-generation',
|
||||
'--selector-membership-file'
|
||||
])
|
||||
const FLAG_OPTIONS = new Set(['--retry-freshness', '--no-monitor-state'])
|
||||
|
||||
// Rejects an unknown option and a repeated one, so a typo can never silently
|
||||
// widen what this check accepts.
|
||||
export function parsePreflightArgs(argv: string[]): {
|
||||
options: Map<string, string>
|
||||
flags: Set<string>
|
||||
} {
|
||||
const args = argv[0] === '--' ? argv.slice(1) : argv
|
||||
const options = new Map<string, string>()
|
||||
const flags = new Set<string>()
|
||||
for (let index = 0; index < args.length; index += 1) {
|
||||
const arg = args[index] as string
|
||||
if (FLAG_OPTIONS.has(arg)) {
|
||||
if (flags.has(arg)) throw new Error(PREFLIGHT_USAGE)
|
||||
flags.add(arg)
|
||||
continue
|
||||
}
|
||||
const value = args[index + 1]
|
||||
if (!VALUE_OPTIONS.has(arg) || options.has(arg) || !value) {
|
||||
throw new Error(PREFLIGHT_USAGE)
|
||||
}
|
||||
options.set(arg, value)
|
||||
index += 1
|
||||
}
|
||||
return { options, flags }
|
||||
}
|
||||
|
||||
// What the live recheck measures the fleet against. Either source supplies the
|
||||
// exact same fields; only where they come from and how they can go stale differs.
|
||||
type PreflightPlan = {
|
||||
environment: 'production'
|
||||
expectedSelector: AdmissionSelector
|
||||
migrationPolicy: 'strict' | 'recover-forward' | 'capacity-transition'
|
||||
recoverySourceCellId: string | null
|
||||
capacityCellId: string | null
|
||||
// The instant the live samples age from. Monitor evidence ages from the moment
|
||||
// the 15-minute window closed; a dispatch plan has no evidence to age, so its
|
||||
// retry budget starts when this process does.
|
||||
evidenceAnchorMs: number
|
||||
}
|
||||
|
||||
async function monitorEvidencePreflightPlan(
|
||||
options: Map<string, string>,
|
||||
nowMs: number,
|
||||
waveIndex: string
|
||||
): Promise<PreflightPlan> {
|
||||
const stateFile = options.get('--state-file')
|
||||
if (
|
||||
!stateFile ||
|
||||
options.has('--expected-selector-generation') ||
|
||||
options.has('--selector-membership-file')
|
||||
) throw new Error(PREFLIGHT_USAGE)
|
||||
const state = PreflightStateSchema.parse(
|
||||
JSON.parse(await readFile(resolve(stateFile), 'utf8'))
|
||||
)
|
||||
const completedAt = Date.parse(state.completedAt)
|
||||
const windowStartedAt = Date.parse(state.windowStartedAt)
|
||||
const lastSampleAt = Date.parse(state.lastSampleAt)
|
||||
const evidenceAgeMs = nowMs - completedAt
|
||||
// Later waves start after sequential predecessor cell rolls, so the
|
||||
// freshness bound grows by one cell-job timeout per predecessor; the live
|
||||
// samples collected below still hold every wave to current health.
|
||||
const maxEvidenceAgeMs =
|
||||
MONITOR_EVIDENCE_MAX_AGE_MS + Number(waveIndex) * WAVE_PREDECESSOR_TIMEOUT_MS
|
||||
if (
|
||||
!preDrainDryRunPassed(state) ||
|
||||
!Number.isFinite(windowStartedAt) ||
|
||||
completedAt - windowStartedAt < 15 * 60_000 ||
|
||||
!Number.isFinite(lastSampleAt) ||
|
||||
lastSampleAt > completedAt ||
|
||||
completedAt - lastSampleAt > state.intervalMs ||
|
||||
!Number.isFinite(completedAt) ||
|
||||
evidenceAgeMs < 0 ||
|
||||
evidenceAgeMs > maxEvidenceAgeMs
|
||||
) {
|
||||
throw new Error('relay live preflight monitor evidence is incomplete or stale')
|
||||
}
|
||||
return {
|
||||
environment: state.environment,
|
||||
expectedSelector: state.expectedSelector,
|
||||
migrationPolicy: state.migrationPolicy,
|
||||
recoverySourceCellId: state.recoverySourceCellId,
|
||||
capacityCellId: state.capacityCellId,
|
||||
evidenceAnchorMs: completedAt
|
||||
}
|
||||
}
|
||||
|
||||
// The expected selector exactly as a same-cap dispatch states it, canonicalised as the monitor CLI
|
||||
// does when it seals evidence. The live selector read from the director is normalised too and the
|
||||
// comparison is an ordered stringify, so unsorted operator input would read as selector drift on a
|
||||
// healthy fleet; normalising is also what enforces every configured cell exactly once.
|
||||
export async function readDispatchSelector(
|
||||
generation: string,
|
||||
membershipFile: string
|
||||
): Promise<AdmissionSelector> {
|
||||
const membership = normalizeSelectorMembership(
|
||||
SelectorMembershipSchema.parse(
|
||||
JSON.parse(await readFile(resolve(membershipFile), 'utf8'))
|
||||
),
|
||||
new Set(relayOpsEnvironment('production').cells.map((cell) => cell.cellId))
|
||||
)
|
||||
return AdmissionSelectorSchema.parse({ generation: Number(generation), membership })
|
||||
}
|
||||
|
||||
// Each predecessor same-cap apply wave reversibly isolates and restores its cell with membership
|
||||
// unchanged (rollback is single-cell, so it never reaches a later wave), so the live selector
|
||||
// comparison must expect the wave-adjusted generation. A general cell advances it by 2; a
|
||||
// migration-only cell is already isolated and stays that way, so its wave advances it by 0. A wave
|
||||
// is never mixed, so one delta covers every predecessor.
|
||||
export function waveAdjustedSelector(
|
||||
selector: AdmissionSelector,
|
||||
waveIndex: string,
|
||||
selectorWaveDelta: string
|
||||
): AdmissionSelector {
|
||||
if (!WAVE_INDEX_PATTERN.test(waveIndex) || !SELECTOR_WAVE_DELTA_PATTERN.test(selectorWaveDelta)) {
|
||||
throw new Error('relay wave index or selector wave delta is invalid')
|
||||
}
|
||||
return {
|
||||
...selector,
|
||||
generation: selector.generation + Number(selectorWaveDelta) * Number(waveIndex)
|
||||
}
|
||||
}
|
||||
|
||||
// No monitor evidence: the expected selector comes straight from the dispatch inputs, which the
|
||||
// rehome inspect after this verifies against the live director. The policy is pinned to strict --
|
||||
// the only one the same-cap rollout ever verifies -- and every threshold is unchanged.
|
||||
async function dispatchPreflightPlan(
|
||||
options: Map<string, string>,
|
||||
nowMs: number
|
||||
): Promise<PreflightPlan> {
|
||||
const generation = options.get('--expected-selector-generation')
|
||||
const membershipFile = options.get('--selector-membership-file')
|
||||
if (!generation || !membershipFile || options.has('--state-file')) {
|
||||
throw new Error(PREFLIGHT_USAGE)
|
||||
}
|
||||
return {
|
||||
environment: 'production',
|
||||
expectedSelector: await readDispatchSelector(generation, membershipFile),
|
||||
migrationPolicy: 'strict',
|
||||
recoverySourceCellId: null,
|
||||
capacityCellId: null,
|
||||
evidenceAnchorMs: nowMs
|
||||
}
|
||||
}
|
||||
|
||||
export async function runIncidentLivePreflight(
|
||||
argv: string[],
|
||||
dependencies: {
|
||||
now?: () => number
|
||||
wait?: (ms: number) => Promise<void>
|
||||
collect?: (expectedSelector: AdmissionSelector) => Promise<IncidentSample>
|
||||
gcloud?: ReturnType<typeof createGcloudClient>
|
||||
environment?: NodeJS.ProcessEnv
|
||||
} = {}
|
||||
): Promise<void> {
|
||||
const parsed = parsePreflightArgs(argv)
|
||||
const waveIndex = parsed.options.get('--wave-index') ?? '0'
|
||||
if (!WAVE_INDEX_PATTERN.test(waveIndex)) throw new Error(PREFLIGHT_USAGE)
|
||||
const selectorWaveDelta = parsed.options.get('--selector-wave-delta') ?? '2'
|
||||
if (!SELECTOR_WAVE_DELTA_PATTERN.test(selectorWaveDelta)) throw new Error(PREFLIGHT_USAGE)
|
||||
const now = dependencies.now ?? Date.now
|
||||
const plan = parsed.flags.has('--no-monitor-state')
|
||||
? await dispatchPreflightPlan(parsed.options, now())
|
||||
: await monitorEvidencePreflightPlan(parsed.options, now(), waveIndex)
|
||||
const maxEvidenceAgeMs =
|
||||
MONITOR_EVIDENCE_MAX_AGE_MS + Number(waveIndex) * WAVE_PREDECESSOR_TIMEOUT_MS
|
||||
const gcloud = livePreflightGcloud(
|
||||
dependencies.gcloud ?? createGcloudClient(),
|
||||
dependencies.environment
|
||||
)
|
||||
const collectOptions = {
|
||||
environment: plan.environment,
|
||||
expectedSelector: waveAdjustedSelector(plan.expectedSelector, waveIndex, selectorWaveDelta),
|
||||
...(dependencies.now ? { now: dependencies.now } : {})
|
||||
}
|
||||
const injected = dependencies.collect
|
||||
const collect = injected
|
||||
? () => injected(collectOptions.expectedSelector)
|
||||
: createIncidentSampleCollector(gcloud, collectOptions)
|
||||
const wait = dependencies.wait ?? ((ms: number) => new Promise<void>((resolveWait) => {
|
||||
setTimeout(resolveWait, ms)
|
||||
}))
|
||||
const freshnessAttempts = parsed.flags.has('--retry-freshness') ? FRESHNESS_RETRY_ATTEMPTS : 1
|
||||
// Why: this single sample decides a mutating wave, so an Asia cell's ~30 s
|
||||
// "no healthy upstream" window, or a one-minute director instance-replacement
|
||||
// dip, could fail a wave here even after the 15-minute gate learned to ride it
|
||||
// out. Hold the two to the same tolerance. Unlike the freshness retry this
|
||||
// needs no flag, because a tolerated breach is never the operator's call to
|
||||
// waive.
|
||||
const cellProbeAttempts = 1 + INCIDENT_MONITOR_THRESHOLDS.cellProbeToleranceSamples
|
||||
const attempts = Math.max(freshnessAttempts, cellProbeAttempts)
|
||||
let freshnessRetries = freshnessAttempts - 1
|
||||
let cellProbeRetries = cellProbeAttempts - 1
|
||||
// Waiting must never carry the mutation past the same evidence-age bound the
|
||||
// entry check enforces, so the wave budget also caps the retry window.
|
||||
const budgetExhausted = (): boolean =>
|
||||
now() + FRESHNESS_RETRY_INTERVAL_MS - plan.evidenceAnchorMs > maxEvidenceAgeMs
|
||||
for (let attempt = 1; attempt <= attempts; attempt++) {
|
||||
// A director admin read can fail on its own (its handler maps a Cloud SQL
|
||||
// pool timeout onto 404), which says nothing about relay health; spend an
|
||||
// attempt on it rather than failing the wave on one unlucky sample.
|
||||
let sample: IncidentSample
|
||||
try {
|
||||
sample = await collect()
|
||||
} catch (error) {
|
||||
const message = error instanceof Error ? error.message : 'sample collection failed'
|
||||
if (attempt === attempts || budgetExhausted()) {
|
||||
throw new Error(`relay live preflight failed: collector: ${message}`)
|
||||
}
|
||||
console.warn(
|
||||
`relay live preflight re-sampling after collector failure (${attempt}/${attempts - 1})`
|
||||
)
|
||||
await wait(FRESHNESS_RETRY_INTERVAL_MS)
|
||||
continue
|
||||
}
|
||||
const evaluation = evaluateIncidentSample(
|
||||
sample,
|
||||
now(),
|
||||
plan.migrationPolicy,
|
||||
plan.recoverySourceCellId,
|
||||
plan.capacityCellId
|
||||
)
|
||||
if (evaluation.status === 'green') return
|
||||
const freshnessFailures = evaluation.failures.filter((failure) =>
|
||||
FRESHNESS_FAILURE_CODES.has(failure.code)
|
||||
)
|
||||
// Tolerated readings only (per-cell probes and the director instance count).
|
||||
// The director and auth health probes are absent here on purpose and fail
|
||||
// the wave on their first bad sample.
|
||||
const cellProbeFailures = evaluation.failures.filter((failure) =>
|
||||
!FRESHNESS_FAILURE_CODES.has(failure.code) && toleratedStreakKey(failure) !== null
|
||||
)
|
||||
const retryable =
|
||||
freshnessFailures.length + cellProbeFailures.length === evaluation.failures.length &&
|
||||
(freshnessFailures.length === 0 || freshnessRetries > 0) &&
|
||||
(cellProbeFailures.length === 0 || cellProbeRetries > 0)
|
||||
if (!retryable || attempt === attempts || budgetExhausted()) {
|
||||
throw new Error(
|
||||
`relay live preflight failed: ${evaluation.failures
|
||||
.map(describeIncidentFailure)
|
||||
.join(',')}`
|
||||
)
|
||||
}
|
||||
if (freshnessFailures.length > 0) freshnessRetries--
|
||||
if (cellProbeFailures.length > 0) cellProbeRetries--
|
||||
console.warn(
|
||||
`relay live preflight re-sampling after tolerable failure (${attempt}/${attempts - 1})`
|
||||
)
|
||||
await wait(FRESHNESS_RETRY_INTERVAL_MS)
|
||||
}
|
||||
}
|
||||
|
||||
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
|
||||
runIncidentLivePreflight(process.argv.slice(2)).catch((error: unknown) => {
|
||||
console.error(error instanceof Error ? error.message : 'relay live preflight failed')
|
||||
process.exitCode = 1
|
||||
})
|
||||
}
|
||||
@@ -1,455 +0,0 @@
|
||||
import { mkdtempSync, readFileSync, rmSync, statSync } from 'node:fs'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { join } from 'node:path'
|
||||
import { afterEach, describe, expect, it } from 'vitest'
|
||||
import {
|
||||
parseIncidentMonitorArguments,
|
||||
runIncidentMonitorCli
|
||||
} from './incident-monitor-cli.js'
|
||||
import type { IncidentSample } from './incident-monitor.js'
|
||||
import { RELAY_OPS_ENVIRONMENTS } from './environment-config.js'
|
||||
|
||||
const directories: string[] = []
|
||||
const startedAt = Date.parse('2026-07-28T00:00:00.000Z')
|
||||
const productionCells = RELAY_OPS_ENVIRONMENTS.production.cells.map((cell) => cell.cellId)
|
||||
const selector = {
|
||||
generation: 1,
|
||||
membership: {
|
||||
existingOnly: productionCells.slice(1),
|
||||
migrationOnly: [],
|
||||
general: [productionCells[0]!]
|
||||
}
|
||||
}
|
||||
const selectorArguments = [
|
||||
'--expected-selector-generation',
|
||||
'1',
|
||||
'--expected-existing-only-cells',
|
||||
productionCells.slice(1).join(','),
|
||||
'--expected-migration-only-cells',
|
||||
'none',
|
||||
'--expected-general-cells',
|
||||
productionCells[0]!
|
||||
]
|
||||
const signal = (value: number, at: number) => ({
|
||||
value,
|
||||
observedAt: new Date(at).toISOString()
|
||||
})
|
||||
|
||||
function sample(at: number): IncidentSample {
|
||||
const observedAt = new Date(at).toISOString()
|
||||
const cellId = 'production-gce-c1'
|
||||
return {
|
||||
collectedAt: observedAt,
|
||||
selector,
|
||||
expectedSelector: selector,
|
||||
cells: [{
|
||||
cellId,
|
||||
region: 'us-central1',
|
||||
runtimeKnown: true,
|
||||
powered: true,
|
||||
expectedAdmissionState: 'general'
|
||||
}],
|
||||
sources: {
|
||||
'active-probe': {
|
||||
observedAt,
|
||||
signals: {
|
||||
'director.health': signal(1, at),
|
||||
'director.ready': signal(1, at),
|
||||
'director.latency_ms': signal(1, at),
|
||||
'auth.health': signal(1, at),
|
||||
'auth.ready': signal(1, at),
|
||||
'auth.latency_ms': signal(1, at),
|
||||
[`cell.${cellId}.health`]: signal(1, at),
|
||||
[`cell.${cellId}.ready`]: signal(1, at),
|
||||
[`cell.${cellId}.latency_ms`]: signal(1, at)
|
||||
}
|
||||
},
|
||||
'cloud-monitoring': {
|
||||
observedAt,
|
||||
signals: {
|
||||
'cloud_sql.cpu': signal(0.1, at),
|
||||
'cloud_sql.memory': signal(0.1, at),
|
||||
'cloud_sql.backends': signal(1, at),
|
||||
'cloud_sql.lock_waits': signal(0, at),
|
||||
'cloud_sql.deadlocks': signal(0, at),
|
||||
'director.instances': signal(5, at),
|
||||
'director.cpu': signal(0.1, at),
|
||||
'director.memory': signal(0.1, at),
|
||||
'director.concurrency': signal(1, at),
|
||||
'director.errors': signal(0, at),
|
||||
'auth.errors': signal(0, at)
|
||||
}
|
||||
},
|
||||
'relay-logs': {
|
||||
observedAt,
|
||||
signals: {
|
||||
'relay.pool_waiting': signal(0, at),
|
||||
'relay.pool_wait_ms': signal(0, at),
|
||||
'relay.postgres_retries': signal(0, at),
|
||||
'relay.postgres_retry_exhausted': signal(0, at),
|
||||
[`cell.${cellId}.connections`]: signal(1, at),
|
||||
[`cell.${cellId}.queued_bytes`]: signal(0, at)
|
||||
}
|
||||
},
|
||||
'director-admin': {
|
||||
observedAt,
|
||||
signals: {
|
||||
[`cell.${cellId}.admission_state`]: signal(2, at),
|
||||
[`cell.${cellId}.heartbeat_fresh`]: signal(1, at),
|
||||
[`cell.${cellId}.heartbeat_age_ms`]: signal(1, at),
|
||||
[`cell.${cellId}.migration_blocked`]: signal(0, at),
|
||||
[`cell.${cellId}.migration_target_inactive`]: signal(0, at)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
afterEach(() => {
|
||||
for (const directory of directories.splice(0)) {
|
||||
rmSync(directory, { recursive: true, force: true })
|
||||
}
|
||||
})
|
||||
|
||||
describe('incident monitor CLI', () => {
|
||||
it('requires an exact selector and rejects invalid membership', () => {
|
||||
expect(() => parseIncidentMonitorArguments([])).toThrow(
|
||||
'--expected-selector-generation'
|
||||
)
|
||||
expect(() =>
|
||||
parseIncidentMonitorArguments([
|
||||
'--incident-id',
|
||||
'incident-1',
|
||||
'--expected-selector-generation',
|
||||
'1',
|
||||
'--expected-existing-only-cells',
|
||||
productionCells.slice(1).join(','),
|
||||
'--expected-migration-only-cells',
|
||||
'none',
|
||||
'--expected-general-cells',
|
||||
'production-gce-c99'
|
||||
])
|
||||
).toThrow('every configured cell exactly once')
|
||||
expect(() =>
|
||||
parseIncidentMonitorArguments([
|
||||
'--incident-id',
|
||||
'incident-1',
|
||||
...selectorArguments,
|
||||
'--interval-seconds',
|
||||
'61'
|
||||
])
|
||||
).toThrow('between 1 and 60')
|
||||
expect(() =>
|
||||
parseIncidentMonitorArguments([
|
||||
'--incident-id',
|
||||
'incident-1',
|
||||
...selectorArguments,
|
||||
'--duration-minutes',
|
||||
'14'
|
||||
])
|
||||
).toThrow('between 15 and 90')
|
||||
expect(() =>
|
||||
parseIncidentMonitorArguments([
|
||||
'--incident-id',
|
||||
'incident-1',
|
||||
'--expected-selector-generation',
|
||||
'0',
|
||||
'--expected-existing-only-cells',
|
||||
productionCells.slice(1).join(','),
|
||||
'--expected-migration-only-cells',
|
||||
productionCells[0]!,
|
||||
'--expected-general-cells',
|
||||
'none'
|
||||
])
|
||||
).toThrow('generation 0 cannot represent migration-only')
|
||||
expect(() =>
|
||||
parseIncidentMonitorArguments([
|
||||
'--incident-id',
|
||||
'incident-1',
|
||||
...selectorArguments,
|
||||
'--migration-policy',
|
||||
'recover-forward',
|
||||
'--pre-drain-dry-run'
|
||||
])
|
||||
).toThrow('requires an existing-only --recovery-source-cell-id')
|
||||
expect(() =>
|
||||
parseIncidentMonitorArguments([
|
||||
'--incident-id',
|
||||
'incident-1',
|
||||
...selectorArguments,
|
||||
'--migration-policy',
|
||||
'capacity-transition',
|
||||
'--pre-drain-dry-run'
|
||||
])
|
||||
).toThrow('requires a general --capacity-cell-id')
|
||||
expect(
|
||||
parseIncidentMonitorArguments([
|
||||
'--incident-id',
|
||||
'incident-1',
|
||||
...selectorArguments,
|
||||
'--migration-policy',
|
||||
'capacity-transition',
|
||||
'--capacity-cell-id',
|
||||
productionCells[0]!,
|
||||
'--pre-drain-dry-run'
|
||||
])
|
||||
).toMatchObject({
|
||||
migrationPolicy: 'capacity-transition',
|
||||
capacityCellId: productionCells[0]!,
|
||||
recoverySourceCellId: null
|
||||
})
|
||||
})
|
||||
|
||||
it('writes private durable checkpoints for a green pre-drain dry run', async () => {
|
||||
const directory = mkdtempSync(join(tmpdir(), 'relay-incident-cli-'))
|
||||
directories.push(directory)
|
||||
let now = startedAt
|
||||
const output: string[] = []
|
||||
const code = await runIncidentMonitorCli(
|
||||
[
|
||||
'--incident-id',
|
||||
'incident-1',
|
||||
...selectorArguments,
|
||||
'--pre-drain-dry-run',
|
||||
'--output-directory',
|
||||
directory
|
||||
],
|
||||
{
|
||||
cwd: directory,
|
||||
now: () => now,
|
||||
wait: async (ms) => {
|
||||
now += ms
|
||||
},
|
||||
collect: async () => sample(now),
|
||||
writeOutput: (value) => output.push(value)
|
||||
}
|
||||
)
|
||||
expect(code).toBe(0)
|
||||
const statePath = join(directory, 'incident-1.state.json')
|
||||
const summaryPath = join(directory, 'incident-1.summaries.jsonl')
|
||||
const markdownPath = join(directory, 'incident-1.summary.md')
|
||||
expect(statSync(statePath).mode & 0o077).toBe(0)
|
||||
expect(statSync(summaryPath).mode & 0o077).toBe(0)
|
||||
expect(statSync(markdownPath).mode & 0o077).toBe(0)
|
||||
const summaries = readFileSync(summaryPath, 'utf8')
|
||||
.trim()
|
||||
.split('\n')
|
||||
.map((line) => JSON.parse(line))
|
||||
expect(summaries.map((entry) => entry.checkpointMinute)).toEqual([0, 5, 15])
|
||||
expect(output.join('')).not.toContain('token')
|
||||
expect(readFileSync(markdownPath, 'utf8')).toContain(
|
||||
'| 0 | 15 | green | 16 | none |'
|
||||
)
|
||||
expect(JSON.parse(readFileSync(statePath, 'utf8'))).toMatchObject({
|
||||
completedAt: new Date(startedAt + 15 * 60_000).toISOString(),
|
||||
frozenAt: null,
|
||||
migrationPolicy: 'strict',
|
||||
recoverySourceCellId: null,
|
||||
capacityCellId: null,
|
||||
sampleCount: 16
|
||||
})
|
||||
})
|
||||
|
||||
it('runs a recovery dry run without masking blocked migrations', async () => {
|
||||
const directory = mkdtempSync(join(tmpdir(), 'relay-incident-cli-'))
|
||||
directories.push(directory)
|
||||
let now = startedAt
|
||||
const recoverySelector = {
|
||||
generation: 1,
|
||||
membership: {
|
||||
existingOnly: ['production-gce-c1'],
|
||||
migrationOnly: [],
|
||||
general: productionCells.slice(1)
|
||||
}
|
||||
}
|
||||
const recoverySample = (): IncidentSample => {
|
||||
const current = sample(now)
|
||||
current.selector = recoverySelector
|
||||
current.expectedSelector = recoverySelector
|
||||
current.cells[0]!.expectedAdmissionState = 'existing-only'
|
||||
current.sources['director-admin']!.signals[
|
||||
'cell.production-gce-c1.admission_state'
|
||||
] = signal(0, now)
|
||||
current.sources['director-admin']!.signals[
|
||||
'cell.production-gce-c1.migration_target_inactive'
|
||||
] = signal(30, now)
|
||||
return current
|
||||
}
|
||||
const args = [
|
||||
'--incident-id',
|
||||
'incident-1',
|
||||
'--expected-selector-generation',
|
||||
'1',
|
||||
'--expected-existing-only-cells',
|
||||
'production-gce-c1',
|
||||
'--expected-migration-only-cells',
|
||||
'none',
|
||||
'--expected-general-cells',
|
||||
productionCells.slice(1).join(','),
|
||||
'--pre-drain-dry-run',
|
||||
'--migration-policy',
|
||||
'recover-forward',
|
||||
'--recovery-source-cell-id',
|
||||
'production-gce-c1',
|
||||
'--output-directory',
|
||||
directory
|
||||
]
|
||||
const dependencies = {
|
||||
cwd: directory,
|
||||
now: () => now,
|
||||
wait: async (ms: number) => {
|
||||
now += ms
|
||||
},
|
||||
collect: async () => recoverySample(),
|
||||
writeOutput: () => {}
|
||||
}
|
||||
await expect(runIncidentMonitorCli(args, dependencies)).resolves.toBe(0)
|
||||
expect(
|
||||
JSON.parse(readFileSync(join(directory, 'incident-1.state.json'), 'utf8'))
|
||||
).toMatchObject({
|
||||
migrationPolicy: 'recover-forward',
|
||||
recoverySourceCellId: 'production-gce-c1',
|
||||
capacityCellId: null,
|
||||
frozenAt: null
|
||||
})
|
||||
})
|
||||
|
||||
it('fails a frozen pre-drain gate after its first sample', async () => {
|
||||
const directory = mkdtempSync(join(tmpdir(), 'relay-incident-cli-'))
|
||||
directories.push(directory)
|
||||
let now = startedAt
|
||||
let collections = 0
|
||||
let waits = 0
|
||||
const code = await runIncidentMonitorCli(
|
||||
[
|
||||
'--incident-id',
|
||||
'incident-1',
|
||||
...selectorArguments,
|
||||
'--pre-drain-dry-run',
|
||||
'--output-directory',
|
||||
directory
|
||||
],
|
||||
{
|
||||
cwd: directory,
|
||||
now: () => now,
|
||||
wait: async (ms) => {
|
||||
waits++
|
||||
now += ms
|
||||
},
|
||||
collect: async () => {
|
||||
collections++
|
||||
const unhealthy = sample(now)
|
||||
unhealthy.sources['relay-logs']!.signals['relay.pool_waiting'] =
|
||||
signal(801, now)
|
||||
return unhealthy
|
||||
},
|
||||
writeOutput: () => {}
|
||||
}
|
||||
)
|
||||
expect(code).toBe(2)
|
||||
expect(collections).toBe(1)
|
||||
expect(waits).toBe(0)
|
||||
expect(
|
||||
JSON.parse(readFileSync(join(directory, 'incident-1.state.json'), 'utf8'))
|
||||
).toMatchObject({
|
||||
completedAt: new Date(startedAt).toISOString(),
|
||||
frozenAt: new Date(startedAt).toISOString(),
|
||||
sampleCount: 1
|
||||
})
|
||||
})
|
||||
|
||||
it('requires --restart and preserves a latched freeze', async () => {
|
||||
const directory = mkdtempSync(join(tmpdir(), 'relay-incident-cli-'))
|
||||
directories.push(directory)
|
||||
let now = startedAt
|
||||
const args = [
|
||||
'--incident-id',
|
||||
'incident-1',
|
||||
...selectorArguments,
|
||||
'--duration-minutes',
|
||||
'15',
|
||||
'--output-directory',
|
||||
directory
|
||||
]
|
||||
await runIncidentMonitorCli(args, {
|
||||
cwd: directory,
|
||||
now: () => now,
|
||||
wait: async (ms) => {
|
||||
now += ms
|
||||
},
|
||||
collect: async () => {
|
||||
const unhealthy = sample(now)
|
||||
unhealthy.sources['relay-logs']!.signals['relay.pool_waiting'] = signal(801, now)
|
||||
return unhealthy
|
||||
},
|
||||
writeOutput: () => {}
|
||||
})
|
||||
await expect(
|
||||
runIncidentMonitorCli(args, {
|
||||
cwd: directory,
|
||||
now: () => now,
|
||||
collect: async () => sample(now),
|
||||
writeOutput: () => {}
|
||||
})
|
||||
).rejects.toThrow('pass --restart')
|
||||
const changedAdmission = [...args]
|
||||
changedAdmission[changedAdmission.indexOf('--expected-selector-generation') + 1] = '2'
|
||||
await expect(
|
||||
runIncidentMonitorCli([...changedAdmission, '--restart'], {
|
||||
cwd: directory,
|
||||
now: () => now,
|
||||
collect: async () => sample(now),
|
||||
writeOutput: () => {}
|
||||
})
|
||||
).rejects.toThrow('do not match')
|
||||
await expect(
|
||||
runIncidentMonitorCli([...args, '--restart'], {
|
||||
cwd: directory,
|
||||
now: () => now,
|
||||
collect: async () => sample(now),
|
||||
writeOutput: () => {}
|
||||
})
|
||||
).resolves.toBe(2)
|
||||
})
|
||||
|
||||
it('resumes a gracefully segmented monitor without resetting continuity', async () => {
|
||||
const directory = mkdtempSync(join(tmpdir(), 'relay-incident-cli-'))
|
||||
directories.push(directory)
|
||||
let now = startedAt
|
||||
const args = [
|
||||
'--incident-id',
|
||||
'incident-1',
|
||||
...selectorArguments,
|
||||
'--duration-minutes',
|
||||
'15',
|
||||
'--output-directory',
|
||||
directory
|
||||
]
|
||||
const dependencies = {
|
||||
cwd: directory,
|
||||
now: () => now,
|
||||
wait: async (ms: number) => {
|
||||
now += ms
|
||||
},
|
||||
collect: async () => sample(now),
|
||||
writeOutput: () => {}
|
||||
}
|
||||
await expect(
|
||||
runIncidentMonitorCli([...args, '--max-samples-this-run', '2'], dependencies)
|
||||
).resolves.toBe(0)
|
||||
const statePath = join(directory, 'incident-1.state.json')
|
||||
expect(JSON.parse(readFileSync(statePath, 'utf8'))).toMatchObject({
|
||||
completedAt: null,
|
||||
sampleCount: 2,
|
||||
lastSampleAt: new Date(startedAt + 60_000).toISOString()
|
||||
})
|
||||
await expect(
|
||||
runIncidentMonitorCli([...args, '--restart'], dependencies)
|
||||
).resolves.toBe(0)
|
||||
expect(JSON.parse(readFileSync(statePath, 'utf8'))).toMatchObject({
|
||||
completedAt: new Date(startedAt + 15 * 60_000).toISOString(),
|
||||
windowSequence: 0,
|
||||
sampleCount: 17
|
||||
})
|
||||
})
|
||||
})
|
||||
@@ -1,510 +0,0 @@
|
||||
import { randomUUID } from 'node:crypto'
|
||||
import {
|
||||
appendFile,
|
||||
chmod,
|
||||
mkdir,
|
||||
open,
|
||||
readFile,
|
||||
rename,
|
||||
stat,
|
||||
writeFile
|
||||
} from 'node:fs/promises'
|
||||
import { dirname, resolve } from 'node:path'
|
||||
import { pathToFileURL } from 'node:url'
|
||||
import { z } from 'zod'
|
||||
import { relayOpsEnvironment } from './environment-config.js'
|
||||
import { createGcloudClient } from './gcloud-client.js'
|
||||
import {
|
||||
AdmissionSelectorSchema,
|
||||
normalizeSelectorMembership,
|
||||
type AdmissionSelector
|
||||
} from './incident-selector.js'
|
||||
import {
|
||||
initialIncidentMonitorState,
|
||||
preDrainDryRunPassed,
|
||||
runIncidentMonitor,
|
||||
type IncidentCheckpoint,
|
||||
type IncidentSample,
|
||||
type IncidentMonitorState
|
||||
} from './incident-monitor.js'
|
||||
import { createIncidentSampleCollector } from './incident-monitor-sources.js'
|
||||
|
||||
const StateSchema = z.object({
|
||||
schemaVersion: z.literal(4),
|
||||
incidentId: z.string(),
|
||||
environment: z.enum(['production', 'staging']),
|
||||
expectedSelector: AdmissionSelectorSchema,
|
||||
preDrainDryRun: z.boolean(),
|
||||
migrationPolicy: z.enum(['strict', 'recover-forward', 'capacity-transition']),
|
||||
recoverySourceCellId: z.string().nullable(),
|
||||
capacityCellId: z.string().nullable(),
|
||||
startedAt: z.string(),
|
||||
windowStartedAt: z.string().nullable(),
|
||||
windowSequence: z.number().int().nonnegative(),
|
||||
durationMinutes: z.number().int(),
|
||||
intervalMs: z.number().int(),
|
||||
nextCheckpointIndex: z.number().int().nonnegative(),
|
||||
sampleCount: z.number().int().nonnegative(),
|
||||
totalSampleCount: z.number().int().nonnegative(),
|
||||
lastSampleAt: z.string().nullable(),
|
||||
continuityEvents: z.array(z.object({
|
||||
recordedAt: z.string(),
|
||||
windowSequence: z.number().int().nonnegative(),
|
||||
// Pre-2026-09-05 state files predate tolerated freshness gaps.
|
||||
tolerated: z.boolean().default(false),
|
||||
failures: z.array(z.object({
|
||||
code: z.string(),
|
||||
source: z.enum(['active-probe', 'cloud-monitoring', 'relay-logs', 'director-admin']),
|
||||
signal: z.string().optional(),
|
||||
observed: z.number().optional(),
|
||||
threshold: z.number().optional()
|
||||
}))
|
||||
})),
|
||||
frozenAt: z.string().nullable(),
|
||||
failures: z.array(z.object({
|
||||
code: z.string(),
|
||||
source: z.enum(['active-probe', 'cloud-monitoring', 'relay-logs', 'director-admin']),
|
||||
signal: z.string().optional(),
|
||||
observed: z.number().optional(),
|
||||
threshold: z.number().optional()
|
||||
})),
|
||||
// Pre-2026-09-17 state files predate cell-probe tolerance; a resumed run that
|
||||
// carries no streak is one that also restarts its window, so it earns nothing.
|
||||
probeStreaks: z.record(z.string(), z.number().int().nonnegative()).default({}),
|
||||
toleratedProbeEvents: z.array(z.object({
|
||||
recordedAt: z.string(),
|
||||
windowSequence: z.number().int().nonnegative(),
|
||||
failures: z.array(z.object({
|
||||
code: z.string(),
|
||||
source: z.enum(['active-probe', 'cloud-monitoring', 'relay-logs', 'director-admin']),
|
||||
signal: z.string().optional(),
|
||||
observed: z.number().optional(),
|
||||
threshold: z.number().optional()
|
||||
}))
|
||||
})).default([]),
|
||||
completedAt: z.string().nullable()
|
||||
})
|
||||
|
||||
type CliOptions = {
|
||||
environment: 'production' | 'staging'
|
||||
incidentId: string
|
||||
durationMinutes: number
|
||||
intervalMs: number
|
||||
expectedSelector: AdmissionSelector
|
||||
stateFile: string
|
||||
summaryFile: string
|
||||
markdownFile: string
|
||||
restart: boolean
|
||||
preDrainDryRun: boolean
|
||||
migrationPolicy: 'strict' | 'recover-forward' | 'capacity-transition'
|
||||
recoverySourceCellId: string | null
|
||||
capacityCellId: string | null
|
||||
maxSamplesThisRun: number | null
|
||||
}
|
||||
|
||||
function parsePositiveInteger(value: string | undefined, name: string): number {
|
||||
const parsed = Number(value)
|
||||
if (!Number.isSafeInteger(parsed) || parsed <= 0) {
|
||||
throw new Error(`${name} must be a positive integer`)
|
||||
}
|
||||
return parsed
|
||||
}
|
||||
|
||||
export function parseIncidentMonitorArguments(argv: string[], cwd = process.cwd()): CliOptions {
|
||||
const flags = new Set(['restart', 'pre-drain-dry-run'])
|
||||
const valueArguments = new Set([
|
||||
'duration-minutes',
|
||||
'environment',
|
||||
'expected-selector-generation',
|
||||
'expected-existing-only-cells',
|
||||
'expected-migration-only-cells',
|
||||
'expected-general-cells',
|
||||
'incident-id',
|
||||
'interval-seconds',
|
||||
'max-samples-this-run',
|
||||
'migration-policy',
|
||||
'output-directory',
|
||||
'recovery-source-cell-id',
|
||||
'capacity-cell-id'
|
||||
])
|
||||
const values: Record<string, string> = {}
|
||||
const enabledFlags = new Set<string>()
|
||||
for (let index = 0; index < argv.length; index++) {
|
||||
const argument = argv[index]
|
||||
if (!argument?.startsWith('--')) throw new Error(`invalid argument ${argument ?? ''}`)
|
||||
const name = argument.slice(2)
|
||||
if (flags.has(name)) {
|
||||
enabledFlags.add(name)
|
||||
continue
|
||||
}
|
||||
if (!valueArguments.has(name)) throw new Error(`unknown argument --${name}`)
|
||||
const value = argv[++index]
|
||||
if (!value || value.startsWith('--')) throw new Error(`missing --${name} value`)
|
||||
values[name] = value
|
||||
}
|
||||
const environment = z.enum(['production', 'staging']).parse(
|
||||
values.environment ?? 'production'
|
||||
)
|
||||
const incidentId = values['incident-id'] ?? randomUUID()
|
||||
if (!/^[a-zA-Z0-9][a-zA-Z0-9._-]{7,127}$/.test(incidentId)) {
|
||||
throw new Error('--incident-id must be 8-128 safe characters')
|
||||
}
|
||||
const selectorGeneration = Number(values['expected-selector-generation'])
|
||||
if (!Number.isSafeInteger(selectorGeneration) || selectorGeneration < 0) {
|
||||
throw new Error('--expected-selector-generation must be a nonnegative integer')
|
||||
}
|
||||
const configuredCells = new Set(
|
||||
relayOpsEnvironment(environment).cells.map((cell) => cell.cellId)
|
||||
)
|
||||
const cellList = (name: string): string[] => {
|
||||
const value = values[name]
|
||||
if (value === undefined) throw new Error(`--${name} is required; use none for an empty set`)
|
||||
return value === 'none' ? [] : value.split(',').map((cellId) => cellId.trim())
|
||||
}
|
||||
const expectedSelector = {
|
||||
generation: selectorGeneration,
|
||||
membership: normalizeSelectorMembership(
|
||||
{
|
||||
existingOnly: cellList('expected-existing-only-cells'),
|
||||
migrationOnly: cellList('expected-migration-only-cells'),
|
||||
general: cellList('expected-general-cells')
|
||||
},
|
||||
configuredCells
|
||||
)
|
||||
}
|
||||
if (selectorGeneration === 0 && expectedSelector.membership.migrationOnly.length > 0) {
|
||||
throw new Error('generation 0 cannot represent migration-only admission')
|
||||
}
|
||||
const preDrainDryRun = enabledFlags.has('pre-drain-dry-run')
|
||||
const migrationPolicy = z.enum([
|
||||
'strict',
|
||||
'recover-forward',
|
||||
'capacity-transition'
|
||||
]).parse(
|
||||
values['migration-policy'] ?? 'strict'
|
||||
)
|
||||
const recoverySourceCellId =
|
||||
values['recovery-source-cell-id'] === undefined ||
|
||||
values['recovery-source-cell-id'] === 'none'
|
||||
? null
|
||||
: values['recovery-source-cell-id']
|
||||
const capacityCellId =
|
||||
values['capacity-cell-id'] === undefined || values['capacity-cell-id'] === 'none'
|
||||
? null
|
||||
: values['capacity-cell-id']
|
||||
const durationMinutes = parsePositiveInteger(
|
||||
values['duration-minutes'] ?? (preDrainDryRun ? '15' : '90'),
|
||||
'--duration-minutes'
|
||||
)
|
||||
const intervalMs =
|
||||
parsePositiveInteger(values['interval-seconds'] ?? '60', '--interval-seconds') * 1_000
|
||||
if (durationMinutes < 15 || durationMinutes > 90) {
|
||||
throw new Error('--duration-minutes must be between 15 and 90')
|
||||
}
|
||||
if (intervalMs > 60_000) {
|
||||
throw new Error('--interval-seconds must be between 1 and 60')
|
||||
}
|
||||
if (preDrainDryRun && durationMinutes !== 15) {
|
||||
throw new Error('--pre-drain-dry-run requires --duration-minutes 15')
|
||||
}
|
||||
if (migrationPolicy !== 'strict' && !preDrainDryRun) {
|
||||
throw new Error(`--migration-policy ${migrationPolicy} requires --pre-drain-dry-run`)
|
||||
}
|
||||
if (
|
||||
migrationPolicy === 'recover-forward' &&
|
||||
(
|
||||
recoverySourceCellId === null ||
|
||||
!configuredCells.has(recoverySourceCellId) ||
|
||||
!expectedSelector.membership.existingOnly.includes(recoverySourceCellId)
|
||||
)
|
||||
) {
|
||||
throw new Error(
|
||||
'--migration-policy recover-forward requires an existing-only --recovery-source-cell-id'
|
||||
)
|
||||
}
|
||||
if (migrationPolicy !== 'recover-forward' && recoverySourceCellId !== null) {
|
||||
throw new Error('--recovery-source-cell-id requires --migration-policy recover-forward')
|
||||
}
|
||||
if (
|
||||
migrationPolicy === 'capacity-transition' &&
|
||||
(
|
||||
capacityCellId === null ||
|
||||
!configuredCells.has(capacityCellId) ||
|
||||
!expectedSelector.membership.general.includes(capacityCellId)
|
||||
)
|
||||
) {
|
||||
throw new Error(
|
||||
'--migration-policy capacity-transition requires a general --capacity-cell-id'
|
||||
)
|
||||
}
|
||||
if (migrationPolicy !== 'capacity-transition' && capacityCellId !== null) {
|
||||
throw new Error('--capacity-cell-id requires --migration-policy capacity-transition')
|
||||
}
|
||||
const directory = resolve(cwd, values['output-directory'] ?? '.relay-incidents')
|
||||
const maxSamplesThisRun = values['max-samples-this-run']
|
||||
? parsePositiveInteger(values['max-samples-this-run'], '--max-samples-this-run')
|
||||
: null
|
||||
return {
|
||||
environment,
|
||||
incidentId,
|
||||
durationMinutes,
|
||||
intervalMs,
|
||||
expectedSelector,
|
||||
stateFile: resolve(directory, `${incidentId}.state.json`),
|
||||
summaryFile: resolve(directory, `${incidentId}.summaries.jsonl`),
|
||||
markdownFile: resolve(directory, `${incidentId}.summary.md`),
|
||||
restart: enabledFlags.has('restart'),
|
||||
preDrainDryRun,
|
||||
migrationPolicy,
|
||||
recoverySourceCellId,
|
||||
capacityCellId,
|
||||
maxSamplesThisRun
|
||||
}
|
||||
}
|
||||
|
||||
async function fileExists(path: string): Promise<boolean> {
|
||||
try {
|
||||
await stat(path)
|
||||
return true
|
||||
} catch {
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
async function syncFile(path: string): Promise<void> {
|
||||
const handle = await open(path, 'r')
|
||||
try {
|
||||
await handle.sync()
|
||||
} finally {
|
||||
await handle.close()
|
||||
}
|
||||
}
|
||||
|
||||
async function persistState(path: string, state: IncidentMonitorState): Promise<void> {
|
||||
await mkdir(dirname(path), { recursive: true, mode: 0o700 })
|
||||
await chmod(dirname(path), 0o700)
|
||||
const temporaryPath = `${path}.tmp`
|
||||
await writeFile(temporaryPath, `${JSON.stringify(state)}\n`, { mode: 0o600 })
|
||||
await chmod(temporaryPath, 0o600)
|
||||
await syncFile(temporaryPath)
|
||||
await rename(temporaryPath, path)
|
||||
await syncFile(path)
|
||||
}
|
||||
|
||||
async function appendCheckpoint(path: string, checkpoint: IncidentCheckpoint): Promise<void> {
|
||||
await mkdir(dirname(path), { recursive: true, mode: 0o700 })
|
||||
await chmod(dirname(path), 0o700)
|
||||
if (await fileExists(path)) {
|
||||
const existing = (await readFile(path, 'utf8'))
|
||||
.trim()
|
||||
.split('\n')
|
||||
.filter(Boolean)
|
||||
.map((line) => JSON.parse(line) as IncidentCheckpoint)
|
||||
if (
|
||||
existing.some((entry) =>
|
||||
entry.windowSequence === checkpoint.windowSequence &&
|
||||
entry.checkpointMinute === checkpoint.checkpointMinute
|
||||
)
|
||||
) return
|
||||
}
|
||||
await appendFile(path, `${JSON.stringify(checkpoint)}\n`, { mode: 0o600 })
|
||||
await chmod(path, 0o600)
|
||||
await syncFile(path)
|
||||
}
|
||||
|
||||
function markdownFailure(checkpoint: IncidentCheckpoint): string {
|
||||
if (checkpoint.failures.length === 0) return 'none'
|
||||
return checkpoint.failures
|
||||
.map((failure) => {
|
||||
const signal = failure.signal ? `/${failure.signal}` : ''
|
||||
return `${failure.source}/${failure.code}${signal}`
|
||||
})
|
||||
.join(', ')
|
||||
}
|
||||
|
||||
async function writeMarkdownSummary(
|
||||
path: string,
|
||||
summaryPath: string,
|
||||
incidentId: string,
|
||||
environment: string
|
||||
): Promise<void> {
|
||||
const checkpoints = (await readFile(summaryPath, 'utf8'))
|
||||
.trim()
|
||||
.split('\n')
|
||||
.filter(Boolean)
|
||||
.map((line) => JSON.parse(line) as IncidentCheckpoint)
|
||||
const rows = checkpoints.map((checkpoint) => [
|
||||
`| ${checkpoint.windowSequence}`,
|
||||
checkpoint.checkpointMinute,
|
||||
checkpoint.status,
|
||||
checkpoint.sampleCount,
|
||||
`${markdownFailure(checkpoint)} |`
|
||||
].join(' | '))
|
||||
const markdown = [
|
||||
'# Relay incident monitor',
|
||||
'',
|
||||
`Incident: \`${incidentId}\``,
|
||||
'',
|
||||
`Environment: \`${environment}\``,
|
||||
'',
|
||||
`Expected selector: \`${JSON.stringify(checkpoints[0]?.expectedSelector ?? null)}\``,
|
||||
'',
|
||||
`Migration policy: \`${checkpoints[0]?.migrationPolicy ?? 'unknown'}\``,
|
||||
'',
|
||||
`Recovery source: \`${checkpoints[0]?.recoverySourceCellId ?? 'none'}\``,
|
||||
'',
|
||||
`Capacity cell: \`${checkpoints[0]?.capacityCellId ?? 'none'}\``,
|
||||
'',
|
||||
'| Window | Minute | Status | Samples | Failures |',
|
||||
'| ---: | ---: | --- | ---: | --- |',
|
||||
...rows,
|
||||
''
|
||||
].join('\n')
|
||||
const temporaryPath = `${path}.tmp`
|
||||
await writeFile(temporaryPath, markdown, { mode: 0o600 })
|
||||
await chmod(temporaryPath, 0o600)
|
||||
await syncFile(temporaryPath)
|
||||
await rename(temporaryPath, path)
|
||||
await syncFile(path)
|
||||
}
|
||||
|
||||
export function suppliedIdentityToken(value: string | undefined): string | null {
|
||||
if (value === undefined) return null
|
||||
if (
|
||||
value.length > 8_192 ||
|
||||
!/^[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+$/.test(value)
|
||||
) {
|
||||
throw new Error('ORCA_RELAY_ADMIN_ID_TOKEN is invalid')
|
||||
}
|
||||
return value
|
||||
}
|
||||
|
||||
async function readInitialState(
|
||||
options: CliOptions,
|
||||
now: () => number = Date.now
|
||||
): Promise<IncidentMonitorState> {
|
||||
const exists = await fileExists(options.stateFile)
|
||||
if (exists && !options.restart) {
|
||||
throw new Error('incident state already exists; pass --restart to resume it')
|
||||
}
|
||||
if (!exists && options.restart) throw new Error('no incident state exists to restart')
|
||||
if (!exists) {
|
||||
return initialIncidentMonitorState({
|
||||
incidentId: options.incidentId,
|
||||
environment: options.environment,
|
||||
expectedSelector: options.expectedSelector,
|
||||
preDrainDryRun: options.preDrainDryRun,
|
||||
migrationPolicy: options.migrationPolicy,
|
||||
recoverySourceCellId: options.recoverySourceCellId,
|
||||
capacityCellId: options.capacityCellId,
|
||||
startedAt: new Date(now()).toISOString(),
|
||||
durationMinutes: options.durationMinutes,
|
||||
intervalMs: options.intervalMs
|
||||
})
|
||||
}
|
||||
const state = StateSchema.parse(
|
||||
JSON.parse(await readFile(options.stateFile, 'utf8'))
|
||||
) as IncidentMonitorState
|
||||
if (
|
||||
state.incidentId !== options.incidentId ||
|
||||
state.environment !== options.environment ||
|
||||
JSON.stringify(state.expectedSelector) !== JSON.stringify(options.expectedSelector) ||
|
||||
state.preDrainDryRun !== options.preDrainDryRun ||
|
||||
state.migrationPolicy !== options.migrationPolicy ||
|
||||
state.recoverySourceCellId !== options.recoverySourceCellId ||
|
||||
state.capacityCellId !== options.capacityCellId ||
|
||||
state.durationMinutes !== options.durationMinutes ||
|
||||
state.intervalMs !== options.intervalMs
|
||||
) {
|
||||
throw new Error('restart arguments do not match durable incident state')
|
||||
}
|
||||
return state
|
||||
}
|
||||
|
||||
export async function runIncidentMonitorCli(
|
||||
argv: string[],
|
||||
dependencies: {
|
||||
cwd?: string
|
||||
now?: () => number
|
||||
wait?: (ms: number) => Promise<void>
|
||||
gcloud?: ReturnType<typeof createGcloudClient>
|
||||
collect?: () => Promise<IncidentSample>
|
||||
writeOutput?: (value: string) => void
|
||||
environment?: NodeJS.ProcessEnv
|
||||
} = {}
|
||||
): Promise<number> {
|
||||
const options = parseIncidentMonitorArguments(argv, dependencies.cwd)
|
||||
const state = await readInitialState(options, dependencies.now)
|
||||
const baseGcloud = dependencies.gcloud ?? createGcloudClient()
|
||||
const token = suppliedIdentityToken(
|
||||
(dependencies.environment ?? process.env).ORCA_RELAY_ADMIN_ID_TOKEN
|
||||
)
|
||||
await persistState(options.stateFile, state)
|
||||
const gcloud = token
|
||||
? { ...baseGcloud, identityToken: async () => token }
|
||||
: baseGcloud
|
||||
const collect =
|
||||
dependencies.collect ??
|
||||
createIncidentSampleCollector(gcloud, {
|
||||
environment: options.environment,
|
||||
expectedSelector: options.expectedSelector,
|
||||
...(dependencies.now ? { now: dependencies.now } : {})
|
||||
})
|
||||
let samplesThisRun = 0
|
||||
const segmentedCollect = async (): Promise<IncidentSample> => {
|
||||
try {
|
||||
return await collect()
|
||||
} finally {
|
||||
samplesThisRun++
|
||||
}
|
||||
}
|
||||
const wait =
|
||||
dependencies.wait ??
|
||||
((ms: number) => new Promise((resolvePromise) => setTimeout(resolvePromise, ms)))
|
||||
const segmentComplete = Symbol('segment-complete')
|
||||
const output = dependencies.writeOutput ?? ((value) => process.stdout.write(`${value}\n`))
|
||||
let result: IncidentMonitorState
|
||||
try {
|
||||
result = await runIncidentMonitor(state, {
|
||||
now: dependencies.now ?? Date.now,
|
||||
wait: async (ms) => {
|
||||
if (
|
||||
options.maxSamplesThisRun !== null &&
|
||||
samplesThisRun >= options.maxSamplesThisRun
|
||||
) {
|
||||
throw segmentComplete
|
||||
}
|
||||
await wait(ms)
|
||||
},
|
||||
collect: segmentedCollect,
|
||||
warn: (message) => console.warn(message),
|
||||
persist: async (nextState) => await persistState(options.stateFile, nextState),
|
||||
checkpoint: async (checkpoint) => {
|
||||
await appendCheckpoint(options.summaryFile, checkpoint)
|
||||
await writeMarkdownSummary(
|
||||
options.markdownFile,
|
||||
options.summaryFile,
|
||||
options.incidentId,
|
||||
options.environment
|
||||
)
|
||||
output(JSON.stringify(checkpoint))
|
||||
}
|
||||
})
|
||||
} catch (error) {
|
||||
if (error !== segmentComplete) throw error
|
||||
return 0
|
||||
}
|
||||
if (options.preDrainDryRun && !preDrainDryRunPassed(result)) return 2
|
||||
return result.frozenAt ? 2 : 0
|
||||
}
|
||||
|
||||
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
|
||||
runIncidentMonitorCli(process.argv.slice(2))
|
||||
.then((code) => {
|
||||
process.exitCode = code
|
||||
})
|
||||
.catch((error: unknown) => {
|
||||
console.error(error instanceof Error ? error.message : 'incident monitor failed')
|
||||
process.exitCode = 1
|
||||
})
|
||||
}
|
||||
@@ -1,655 +0,0 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { RELAY_OPS_ENVIRONMENTS } from './environment-config.js'
|
||||
import type { GcloudClient } from './gcloud-client.js'
|
||||
import { effectiveAdmissionState } from './incident-selector.js'
|
||||
import { INCIDENT_MONITOR_THRESHOLDS } from './incident-monitor.js'
|
||||
import {
|
||||
directorSignals,
|
||||
GOOGLE_METRICS,
|
||||
readGoogleMetric,
|
||||
readGoogleMetricWithEmptyRetry,
|
||||
relayFiveMinuteDeltaSignal
|
||||
} from './incident-monitor-sources.js'
|
||||
|
||||
const now = Date.parse('2026-07-28T10:00:00.000Z')
|
||||
const startAt = new Date(now - 5 * 60_000).toISOString()
|
||||
const endAt = new Date(now).toISOString()
|
||||
const productionCells = RELAY_OPS_ENVIRONMENTS.production.cells.map(
|
||||
({ cellId }) => cellId
|
||||
)
|
||||
|
||||
describe('incident monitor sources', () => {
|
||||
it('uses legacy booleans only at selector generation zero', () => {
|
||||
const membership = {
|
||||
existingOnly: ['c1'],
|
||||
migrationOnly: [],
|
||||
general: ['c2']
|
||||
}
|
||||
expect(
|
||||
effectiveAdmissionState({ generation: 0, membership }, true, 'c1')
|
||||
).toBe('general')
|
||||
expect(
|
||||
effectiveAdmissionState({ generation: 1, membership }, true, 'c1')
|
||||
).toBe('existing-only')
|
||||
})
|
||||
|
||||
it('sums DELTA metrics across the window and scopes every target exactly', async () => {
|
||||
const filters: string[] = []
|
||||
const fetchImpl: typeof fetch = async (input) => {
|
||||
const url = new URL(String(input))
|
||||
filters.push(url.searchParams.get('filter') ?? '')
|
||||
return Response.json({
|
||||
timeSeries: [
|
||||
{
|
||||
points: [
|
||||
{
|
||||
interval: { endTime: new Date(now - 120_000).toISOString() },
|
||||
value: { int64Value: '2' }
|
||||
},
|
||||
{
|
||||
interval: { endTime: new Date(now - 60_000).toISOString() },
|
||||
value: { int64Value: '3' }
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
})
|
||||
}
|
||||
const environment = RELAY_OPS_ENVIRONMENTS.production
|
||||
const directorErrors = GOOGLE_METRICS.find(
|
||||
(definition) => definition.signal === 'director.errors'
|
||||
)!
|
||||
const deadlocks = GOOGLE_METRICS.find(
|
||||
(definition) => definition.signal === 'cloud_sql.deadlocks'
|
||||
)!
|
||||
await expect(
|
||||
readGoogleMetric(
|
||||
environment,
|
||||
directorErrors,
|
||||
'secret-access-token',
|
||||
startAt,
|
||||
endAt,
|
||||
fetchImpl
|
||||
)
|
||||
).resolves.toEqual({
|
||||
value: 5,
|
||||
observedAt: new Date(now - 60_000).toISOString()
|
||||
})
|
||||
await readGoogleMetric(
|
||||
environment,
|
||||
deadlocks,
|
||||
'secret-access-token',
|
||||
startAt,
|
||||
endAt,
|
||||
fetchImpl
|
||||
)
|
||||
expect(filters[0]).toContain(
|
||||
'resource.label."service_name"="orca-cloud-relay"'
|
||||
)
|
||||
expect(filters[0]).toContain('metric.label."response_code"!="503"')
|
||||
expect(filters[1]).toContain(
|
||||
'resource.label."database_id"="onorca-cloud:orca-cloud-auth-db"'
|
||||
)
|
||||
})
|
||||
|
||||
it('zero-fills an expired sparse lock-wait point', async () => {
|
||||
let pointAt = now - INCIDENT_MONITOR_THRESHOLDS.cloudLockWaitCarryMs
|
||||
const fetchImpl: typeof fetch = async () => Response.json({
|
||||
timeSeries: [{
|
||||
points: [{
|
||||
interval: { endTime: new Date(pointAt).toISOString() },
|
||||
value: { int64Value: '1' }
|
||||
}]
|
||||
}]
|
||||
})
|
||||
const definition = GOOGLE_METRICS.find(
|
||||
({ signal }) => signal === 'cloud_sql.lock_waits'
|
||||
)!
|
||||
await expect(readGoogleMetric(
|
||||
RELAY_OPS_ENVIRONMENTS.production,
|
||||
definition,
|
||||
'secret-access-token',
|
||||
startAt,
|
||||
endAt,
|
||||
fetchImpl
|
||||
)).resolves.toEqual({
|
||||
value: 1,
|
||||
observedAt: new Date(pointAt).toISOString()
|
||||
})
|
||||
await expect(readGoogleMetric(
|
||||
RELAY_OPS_ENVIRONMENTS.production,
|
||||
definition,
|
||||
'secret-access-token',
|
||||
startAt,
|
||||
endAt,
|
||||
fetchImpl,
|
||||
() => now + 3_000
|
||||
)).resolves.toEqual({
|
||||
value: 1,
|
||||
observedAt: new Date(pointAt).toISOString()
|
||||
})
|
||||
pointAt--
|
||||
await expect(readGoogleMetric(
|
||||
RELAY_OPS_ENVIRONMENTS.production,
|
||||
definition,
|
||||
'secret-access-token',
|
||||
startAt,
|
||||
endAt,
|
||||
fetchImpl
|
||||
)).resolves.toEqual({ value: 0, observedAt: endAt })
|
||||
})
|
||||
|
||||
it('freshens a sparse zero without masking a recent nonzero lock wait', async () => {
|
||||
let value = 0
|
||||
const pointAt = now - INCIDENT_MONITOR_THRESHOLDS.cloudLockWaitCarryMs
|
||||
const readAt = now + 11_879
|
||||
const fetchImpl: typeof fetch = async () => Response.json({
|
||||
timeSeries: [{
|
||||
points: [{
|
||||
interval: { endTime: new Date(pointAt).toISOString() },
|
||||
value: { int64Value: String(value) }
|
||||
}]
|
||||
}]
|
||||
})
|
||||
const definition = GOOGLE_METRICS.find(
|
||||
({ signal }) => signal === 'cloud_sql.lock_waits'
|
||||
)!
|
||||
|
||||
const sparseZero = await readGoogleMetric(
|
||||
RELAY_OPS_ENVIRONMENTS.production,
|
||||
definition,
|
||||
'secret-access-token',
|
||||
startAt,
|
||||
endAt,
|
||||
fetchImpl,
|
||||
() => readAt
|
||||
)
|
||||
expect(sparseZero).toEqual({
|
||||
value: 0,
|
||||
observedAt: new Date(readAt).toISOString()
|
||||
})
|
||||
expect(readAt - pointAt).toBe(191_879)
|
||||
expect(readAt - Date.parse(sparseZero!.observedAt)).toBe(0)
|
||||
|
||||
value = 20
|
||||
await expect(readGoogleMetric(
|
||||
RELAY_OPS_ENVIRONMENTS.production,
|
||||
definition,
|
||||
'secret-access-token',
|
||||
startAt,
|
||||
endAt,
|
||||
fetchImpl,
|
||||
() => readAt
|
||||
)).resolves.toEqual({
|
||||
value: 20,
|
||||
observedAt: new Date(pointAt).toISOString()
|
||||
})
|
||||
})
|
||||
|
||||
it('preserves a recent nonzero lock wait across staggered series', async () => {
|
||||
const nonzeroAt = now - 179_000
|
||||
const zeroAt = now - 178_000
|
||||
const definition = GOOGLE_METRICS.find(
|
||||
({ signal }) => signal === 'cloud_sql.lock_waits'
|
||||
)!
|
||||
const metric = await readGoogleMetric(
|
||||
RELAY_OPS_ENVIRONMENTS.production,
|
||||
definition,
|
||||
'secret-access-token',
|
||||
startAt,
|
||||
endAt,
|
||||
async () => Response.json({
|
||||
timeSeries: [
|
||||
{
|
||||
points: [{
|
||||
interval: { endTime: new Date(nonzeroAt).toISOString() },
|
||||
value: { int64Value: '7' }
|
||||
}]
|
||||
},
|
||||
{
|
||||
points: [{
|
||||
interval: { endTime: new Date(zeroAt).toISOString() },
|
||||
value: { int64Value: '0' }
|
||||
}]
|
||||
}
|
||||
]
|
||||
})
|
||||
)
|
||||
|
||||
expect(metric).toEqual({
|
||||
value: 7,
|
||||
observedAt: new Date(nonzeroAt).toISOString()
|
||||
})
|
||||
|
||||
await expect(readGoogleMetric(
|
||||
RELAY_OPS_ENVIRONMENTS.production,
|
||||
definition,
|
||||
'secret-access-token',
|
||||
startAt,
|
||||
endAt,
|
||||
async () => Response.json({
|
||||
timeSeries: [{
|
||||
points: [
|
||||
{
|
||||
interval: { endTime: new Date(nonzeroAt).toISOString() },
|
||||
value: { int64Value: '7' }
|
||||
},
|
||||
{
|
||||
interval: { endTime: new Date(zeroAt).toISOString() },
|
||||
value: { int64Value: '0' }
|
||||
}
|
||||
]
|
||||
}]
|
||||
})
|
||||
)).resolves.toEqual({ value: 0, observedAt: endAt })
|
||||
})
|
||||
|
||||
it('retries an empty required metric without weakening its value', async () => {
|
||||
let calls = 0
|
||||
const waits: number[] = []
|
||||
const definition = GOOGLE_METRICS.find(
|
||||
({ signal }) => signal === 'cloud_sql.cpu'
|
||||
)!
|
||||
await expect(readGoogleMetricWithEmptyRetry(
|
||||
RELAY_OPS_ENVIRONMENTS.production,
|
||||
definition,
|
||||
'secret-access-token',
|
||||
startAt,
|
||||
endAt,
|
||||
async () => Response.json(calls++ === 0
|
||||
? { timeSeries: [] }
|
||||
: {
|
||||
timeSeries: [{
|
||||
points: [{
|
||||
interval: { endTime: new Date(now - 60_000).toISOString() },
|
||||
value: { doubleValue: 0.81 }
|
||||
}]
|
||||
}]
|
||||
}),
|
||||
() => now,
|
||||
async (ms) => { waits.push(ms) }
|
||||
)).resolves.toEqual({
|
||||
value: 0.81,
|
||||
observedAt: new Date(now - 60_000).toISOString()
|
||||
})
|
||||
expect(calls).toBe(2)
|
||||
expect(waits).toEqual([2_000])
|
||||
})
|
||||
|
||||
it('still reports a required metric missing after bounded retries', async () => {
|
||||
let calls = 0
|
||||
const waits: number[] = []
|
||||
const definition = GOOGLE_METRICS.find(
|
||||
({ signal }) => signal === 'director.concurrency'
|
||||
)!
|
||||
await expect(readGoogleMetricWithEmptyRetry(
|
||||
RELAY_OPS_ENVIRONMENTS.production,
|
||||
definition,
|
||||
'secret-access-token',
|
||||
startAt,
|
||||
endAt,
|
||||
async () => {
|
||||
calls++
|
||||
return Response.json({ timeSeries: [] })
|
||||
},
|
||||
() => now,
|
||||
async (ms) => { waits.push(ms) }
|
||||
)).resolves.toBeNull()
|
||||
expect(calls).toBe(3)
|
||||
expect(waits).toEqual([2_000, 2_000])
|
||||
})
|
||||
|
||||
it('timestamps sparse retry aggregates at query completion', () => {
|
||||
expect(relayFiveMinuteDeltaSignal({
|
||||
available: true,
|
||||
points: [
|
||||
{ at: new Date(now - 22 * 60_000).toISOString(), value: 7 },
|
||||
{ at: new Date(now - 4 * 60_000).toISOString(), value: 2 }
|
||||
]
|
||||
}, endAt)).toEqual({ value: 2, observedAt: endAt })
|
||||
expect(relayFiveMinuteDeltaSignal({
|
||||
available: true,
|
||||
points: [{ at: new Date(now - 22 * 60_000).toISOString(), value: 7 }]
|
||||
}, endAt)).toEqual({ value: 0, observedAt: endAt })
|
||||
expect(relayFiveMinuteDeltaSignal({
|
||||
available: false,
|
||||
points: []
|
||||
}, endAt)).toBeNull()
|
||||
})
|
||||
|
||||
// Why: the per-region cell latency bar is only correct if the tfvars region
|
||||
// reaches the evaluator on every cell expectation.
|
||||
it('carries the configured region onto every cell expectation', async () => {
|
||||
const gcloud: GcloudClient = {
|
||||
accessToken: async () => 'unused',
|
||||
identityToken: async () => 'unused'
|
||||
}
|
||||
const selector = {
|
||||
generation: 1,
|
||||
membership: {
|
||||
existingOnly: [],
|
||||
migrationOnly: [],
|
||||
general: productionCells
|
||||
}
|
||||
}
|
||||
const fetchImpl: typeof fetch = async (_input, init) => {
|
||||
const body = JSON.parse(String(init?.body)) as { cellId?: string; sourceCellId?: string }
|
||||
if (!body.cellId && !body.sourceCellId) return Response.json({ selector })
|
||||
if (body.cellId) {
|
||||
return Response.json({
|
||||
status: {
|
||||
enabled: true,
|
||||
connectionCapacity: { hardCap: 600 },
|
||||
runtime: { lastHeartbeatAt: now - 1_000, heartbeatFresh: true }
|
||||
}
|
||||
})
|
||||
}
|
||||
return Response.json({
|
||||
blocked: 0,
|
||||
blockedExpiredUnregistered: 0,
|
||||
registeredTargetInactive: 0
|
||||
})
|
||||
}
|
||||
const result = await directorSignals('production', selector, gcloud, now, fetchImpl)
|
||||
const regionById = new Map(result.cells.map((cell) => [cell.cellId, cell.region]))
|
||||
expect(regionById.get('production-gce-c1')).toBe('us-central1')
|
||||
expect(regionById.get('production-gce-c27')).toBe('asia-east2')
|
||||
expect(result.cells).toHaveLength(productionCells.length)
|
||||
for (const cell of RELAY_OPS_ENVIRONMENTS.production.cells) {
|
||||
expect(regionById.get(cell.cellId)).toBe(cell.region)
|
||||
}
|
||||
})
|
||||
|
||||
it('aggregates admin state without returning tokens or response identities', async () => {
|
||||
const identityToken = 'secret.header.signature'
|
||||
const sensitiveIdentity = 'user@example.test'
|
||||
const gcloud: GcloudClient = {
|
||||
accessToken: async () => 'unused',
|
||||
identityToken: async () => identityToken
|
||||
}
|
||||
let activeRequests = 0
|
||||
let maximumActiveRequests = 0
|
||||
let requestCount = 0
|
||||
const fetchImpl: typeof fetch = async (_input, init) => {
|
||||
requestCount++
|
||||
activeRequests++
|
||||
maximumActiveRequests = Math.max(maximumActiveRequests, activeRequests)
|
||||
expect(new Headers(init?.headers).get('authorization')).toBe(
|
||||
`Bearer ${identityToken}`
|
||||
)
|
||||
const body = JSON.parse(String(init?.body)) as {
|
||||
cellId?: string
|
||||
sourceCellId?: string
|
||||
targetCellId?: string
|
||||
}
|
||||
await Promise.resolve()
|
||||
activeRequests--
|
||||
if (!body.cellId && !body.sourceCellId) {
|
||||
return Response.json({
|
||||
selector: {
|
||||
generation: 1,
|
||||
membership: {
|
||||
existingOnly: productionCells.slice(2),
|
||||
migrationOnly: ['production-gce-c2'],
|
||||
general: ['production-gce-c1']
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
if (body.cellId) {
|
||||
return Response.json({
|
||||
status: {
|
||||
// Selector-era monitoring must ignore this legacy compatibility bit.
|
||||
enabled: body.cellId !== 'production-gce-c1',
|
||||
connectionCapacity: {
|
||||
hardCap: body.cellId === 'production-gce-c1' ? 1_000 : 600
|
||||
},
|
||||
runtime: {
|
||||
lastHeartbeatAt: now - 1_000,
|
||||
heartbeatFresh: true
|
||||
},
|
||||
userId: sensitiveIdentity
|
||||
}
|
||||
})
|
||||
}
|
||||
return Response.json({
|
||||
blocked: 0,
|
||||
blockedExpiredUnregistered:
|
||||
body.sourceCellId === 'production-gce-c1' &&
|
||||
body.targetCellId === 'production-gce-c2'
|
||||
? 1
|
||||
: 0,
|
||||
registeredTargetInactive: 0,
|
||||
userId: sensitiveIdentity
|
||||
})
|
||||
}
|
||||
const result = await directorSignals(
|
||||
'production',
|
||||
{
|
||||
generation: 1,
|
||||
membership: {
|
||||
existingOnly: productionCells.slice(2),
|
||||
migrationOnly: ['production-gce-c2'],
|
||||
general: ['production-gce-c1']
|
||||
}
|
||||
},
|
||||
gcloud,
|
||||
now,
|
||||
fetchImpl
|
||||
)
|
||||
expect(requestCount).toBe(productionCells.length * 2)
|
||||
expect(maximumActiveRequests).toBe(1)
|
||||
expect(
|
||||
result.source.signals['cell.production-gce-c1.migration_blocked']
|
||||
).toMatchObject({ value: 1 })
|
||||
expect(result.cells.find((cell) => cell.cellId === 'production-gce-c1')).toMatchObject({
|
||||
expectedAdmissionState: 'general'
|
||||
})
|
||||
expect(
|
||||
result.source.signals['cell.production-gce-c1.admission_state']
|
||||
).toMatchObject({ value: 2 })
|
||||
expect(
|
||||
result.source.signals['cell.production-gce-c1.connection_hard_cap']
|
||||
).toMatchObject({ value: 1_000 })
|
||||
expect(
|
||||
result.source.signals['cell.production-gce-c2.admission_state']
|
||||
).toMatchObject({ value: 1 })
|
||||
const serialized = JSON.stringify(result)
|
||||
expect(serialized).not.toContain(identityToken)
|
||||
expect(serialized).not.toContain(sensitiveIdentity)
|
||||
})
|
||||
|
||||
// Why: the director maps a Cloud SQL pool connect timeout in cell-status onto
|
||||
// a 404, so without this the whole sample dies on one 2 s database stall.
|
||||
it('retries a transient director admin failure and then reports the cell', async () => {
|
||||
const gcloud: GcloudClient = {
|
||||
accessToken: async () => 'unused',
|
||||
identityToken: async () => 'unused'
|
||||
}
|
||||
const selector = {
|
||||
generation: 1,
|
||||
membership: { existingOnly: [], migrationOnly: [], general: productionCells }
|
||||
}
|
||||
const waits: number[] = []
|
||||
let cellStatusCalls = 0
|
||||
const fetchImpl: typeof fetch = async (_input, init) => {
|
||||
const body = JSON.parse(String(init?.body)) as { cellId?: string; sourceCellId?: string }
|
||||
if (!body.cellId && !body.sourceCellId) return Response.json({ selector })
|
||||
if (body.cellId) {
|
||||
cellStatusCalls++
|
||||
if (cellStatusCalls === 1) {
|
||||
return Response.json(
|
||||
{ error: 'timeout exceeded when trying to connect' },
|
||||
{ status: 404 }
|
||||
)
|
||||
}
|
||||
return Response.json({
|
||||
status: {
|
||||
enabled: true,
|
||||
connectionCapacity: { hardCap: 600 },
|
||||
runtime: { lastHeartbeatAt: now - 1_000, heartbeatFresh: true }
|
||||
}
|
||||
})
|
||||
}
|
||||
return Response.json({
|
||||
blocked: 0,
|
||||
blockedExpiredUnregistered: 0,
|
||||
registeredTargetInactive: 0
|
||||
})
|
||||
}
|
||||
const result = await directorSignals(
|
||||
'production',
|
||||
selector,
|
||||
gcloud,
|
||||
now,
|
||||
fetchImpl,
|
||||
async (ms) => {
|
||||
waits.push(ms)
|
||||
}
|
||||
)
|
||||
expect(waits).toEqual([5_000])
|
||||
expect(cellStatusCalls).toBe(productionCells.length + 1)
|
||||
expect(result.cells).toHaveLength(productionCells.length)
|
||||
expect(
|
||||
result.source.signals['cell.production-gce-c1.connection_hard_cap']
|
||||
).toMatchObject({ value: 600 })
|
||||
})
|
||||
|
||||
// A rejected admin token is a decision, not weather: retrying it only burns
|
||||
// the sample budget and hides the misconfiguration.
|
||||
it('fails immediately on an unauthorized director admin response', async () => {
|
||||
const gcloud: GcloudClient = {
|
||||
accessToken: async () => 'unused',
|
||||
identityToken: async () => 'unused'
|
||||
}
|
||||
const selector = {
|
||||
generation: 1,
|
||||
membership: { existingOnly: [], migrationOnly: [], general: productionCells }
|
||||
}
|
||||
const waits: number[] = []
|
||||
let requestCount = 0
|
||||
const fetchImpl: typeof fetch = async (_input, init) => {
|
||||
requestCount++
|
||||
const body = JSON.parse(String(init?.body)) as { cellId?: string; sourceCellId?: string }
|
||||
if (!body.cellId && !body.sourceCellId) return Response.json({ selector })
|
||||
return Response.json({ error: 'invalid_token' }, { status: 401 })
|
||||
}
|
||||
await expect(
|
||||
directorSignals('production', selector, gcloud, now, fetchImpl, async (ms) => {
|
||||
waits.push(ms)
|
||||
})
|
||||
).rejects.toThrow('Relay admin telemetry returned 401')
|
||||
expect(requestCount).toBe(2)
|
||||
expect(waits).toEqual([])
|
||||
})
|
||||
|
||||
// Why: on 2026-10-19 cell-status refused reserve production-gce-c7 with a 404 on every sample.
|
||||
it('marks a reserve cell unprobed and skips its refused migration reads', async () => {
|
||||
const gcloud: GcloudClient = {
|
||||
accessToken: async () => 'unused',
|
||||
identityToken: async () => 'unused'
|
||||
}
|
||||
const reserveCellId = 'production-gce-c7'
|
||||
const selector = {
|
||||
generation: 1,
|
||||
membership: {
|
||||
existingOnly: [],
|
||||
migrationOnly: ['production-gce-c2'],
|
||||
general: productionCells.filter((cellId) => cellId !== 'production-gce-c2')
|
||||
}
|
||||
}
|
||||
const waits: number[] = []
|
||||
const fetchImpl: typeof fetch = async (input, init) => {
|
||||
const path = new URL(String(input)).pathname
|
||||
const body = JSON.parse(String(init?.body)) as Record<string, unknown>
|
||||
if (path === '/v1/admin/admission-selector/status') return Response.json({ selector })
|
||||
if (path === '/v1/admin/cell-status') {
|
||||
if (body.cellId === reserveCellId) {
|
||||
return Response.json({ error: 'cell_in_reserve_mode' }, { status: 404 })
|
||||
}
|
||||
return Response.json({
|
||||
status: {
|
||||
enabled: true,
|
||||
connectionCapacity: { hardCap: 600 },
|
||||
runtime: { lastHeartbeatAt: now - 1_000, heartbeatFresh: true }
|
||||
}
|
||||
})
|
||||
}
|
||||
expect(path).toBe('/v1/admin/evacuation-status')
|
||||
expect(body.sourceCellId).not.toBe(reserveCellId)
|
||||
return Response.json({
|
||||
blocked: 0,
|
||||
blockedExpiredUnregistered: 0,
|
||||
registeredTargetInactive: 0
|
||||
})
|
||||
}
|
||||
const result = await directorSignals('production', selector, gcloud, now, fetchImpl, async (ms) => {
|
||||
waits.push(ms)
|
||||
})
|
||||
expect(waits).toEqual([])
|
||||
expect(result.cells.find((cell) => cell.cellId === reserveCellId)).toMatchObject({
|
||||
reserve: true
|
||||
})
|
||||
expect(result.cells.find((cell) => cell.cellId === 'production-gce-c1')).not.toHaveProperty(
|
||||
'reserve'
|
||||
)
|
||||
const signals = result.source.signals
|
||||
expect(signals[`cell.${reserveCellId}.admission_state`]).toMatchObject({ value: 2 })
|
||||
// Why: 10-19 run 39225932592 froze reserve c7 at 731 connections against the 500 default.
|
||||
const configuredCap = RELAY_OPS_ENVIRONMENTS.production.cells.find(
|
||||
(cell) => cell.cellId === reserveCellId
|
||||
)?.connectionHardCap
|
||||
expect(configuredCap).toBe(1_000)
|
||||
expect(signals[`cell.${reserveCellId}.connection_hard_cap`]).toMatchObject({
|
||||
value: configuredCap
|
||||
})
|
||||
for (const suffix of ['heartbeat_fresh', 'heartbeat_age_ms', 'migration_blocked']) {
|
||||
expect(signals[`cell.${reserveCellId}.${suffix}`]).toBeUndefined()
|
||||
}
|
||||
expect(signals['cell.production-gce-c1.heartbeat_age_ms']).toMatchObject({ value: 1_000 })
|
||||
})
|
||||
|
||||
it('still fails a cell-status 404 that is not the reserve refusal', async () => {
|
||||
const gcloud: GcloudClient = {
|
||||
accessToken: async () => 'unused',
|
||||
identityToken: async () => 'unused'
|
||||
}
|
||||
const selector = {
|
||||
generation: 1,
|
||||
membership: { existingOnly: [], migrationOnly: [], general: productionCells }
|
||||
}
|
||||
const collect = (error: string) =>
|
||||
directorSignals('production', selector, gcloud, now, async (input) => {
|
||||
const path = new URL(String(input)).pathname
|
||||
if (path === '/v1/admin/admission-selector/status') return Response.json({ selector })
|
||||
return Response.json({ error }, { status: 404 })
|
||||
}, async () => {})
|
||||
for (const error of ['cell_not_found', 'cell_in_reserve_mode_soon', '']) {
|
||||
await expect(collect(error)).rejects.toThrow('Relay admin telemetry returned 404')
|
||||
}
|
||||
})
|
||||
|
||||
// A 404 the director means (wrong role) must not be retried either.
|
||||
it('does not retry a director-only 404', async () => {
|
||||
const gcloud: GcloudClient = {
|
||||
accessToken: async () => 'unused',
|
||||
identityToken: async () => 'unused'
|
||||
}
|
||||
let requestCount = 0
|
||||
const fetchImpl: typeof fetch = async () => {
|
||||
requestCount++
|
||||
return Response.json({ error: 'director_only' }, { status: 404 })
|
||||
}
|
||||
await expect(
|
||||
directorSignals(
|
||||
'production',
|
||||
{ generation: 1, membership: { existingOnly: [], migrationOnly: [], general: productionCells } },
|
||||
gcloud,
|
||||
now,
|
||||
fetchImpl,
|
||||
async () => {}
|
||||
)
|
||||
).rejects.toThrow('Relay admin telemetry returned 404')
|
||||
expect(requestCount).toBe(1)
|
||||
})
|
||||
})
|
||||
@@ -1,783 +0,0 @@
|
||||
import { z } from 'zod'
|
||||
import { buildDashboardSnapshot } from './dashboard-snapshot.js'
|
||||
import type {
|
||||
RelayOpsEnvironment,
|
||||
RelayOpsEnvironmentId
|
||||
} from './environment-config.js'
|
||||
import { relayOpsEnvironment } from './environment-config.js'
|
||||
import type { GcloudClient } from './gcloud-client.js'
|
||||
import type { RelayMetricSnapshot } from './monitoring-snapshot.js'
|
||||
import {
|
||||
AdmissionSelectorSchema,
|
||||
effectiveAdmissionState,
|
||||
normalizeSelectorMembership,
|
||||
selectorCellState,
|
||||
type AdmissionSelector,
|
||||
} from './incident-selector.js'
|
||||
import {
|
||||
INCIDENT_MONITOR_THRESHOLDS,
|
||||
type IncidentSample,
|
||||
type IncidentSignal,
|
||||
type IncidentSource
|
||||
} from './incident-monitor.js'
|
||||
|
||||
const NumericSchema = z.union([z.number(), z.string()])
|
||||
.transform(Number)
|
||||
.pipe(z.number().finite())
|
||||
export const MonitoringPointSchema = z.object({
|
||||
interval: z.object({ endTime: z.string() }),
|
||||
value: z.object({
|
||||
doubleValue: NumericSchema.optional(),
|
||||
int64Value: NumericSchema.optional(),
|
||||
distributionValue: z.object({
|
||||
mean: NumericSchema.optional(),
|
||||
range: z.object({ max: NumericSchema.optional() }).optional()
|
||||
}).optional()
|
||||
})
|
||||
})
|
||||
export const MonitoringResponseSchema = z.object({
|
||||
timeSeries: z.array(z.object({ points: z.array(MonitoringPointSchema) })).default([]),
|
||||
nextPageToken: z.string().optional()
|
||||
})
|
||||
const CellStatusSchema = z.object({
|
||||
status: z.object({
|
||||
enabled: z.boolean(),
|
||||
connectionCapacity: z
|
||||
.object({ hardCap: z.number().int().positive() })
|
||||
.nullable(),
|
||||
runtime: z.object({
|
||||
lastHeartbeatAt: z.number(),
|
||||
heartbeatFresh: z.boolean()
|
||||
}).nullable()
|
||||
})
|
||||
})
|
||||
const SelectorStatusSchema = z.object({
|
||||
selector: AdmissionSelectorSchema
|
||||
})
|
||||
const MigrationStatusSchema = z.object({
|
||||
blocked: z.number().int().nonnegative(),
|
||||
registeredTargetInactive: z.number().int().nonnegative(),
|
||||
blockedExpiredUnregistered: z.number().int().nonnegative()
|
||||
})
|
||||
|
||||
export type GoogleMetricDefinition = {
|
||||
signal: string
|
||||
type: string
|
||||
resourceFilter: string
|
||||
aggregation: 'latest-max' | 'latest-sum' | 'window-sum'
|
||||
emptyIsZero?: boolean
|
||||
zeroAfterMs?: number
|
||||
}
|
||||
|
||||
export const GOOGLE_METRICS: GoogleMetricDefinition[] = [
|
||||
{
|
||||
signal: 'cloud_sql.cpu',
|
||||
type: 'cloudsql.googleapis.com/database/cpu/utilization',
|
||||
resourceFilter: 'resource.type="cloudsql_database"',
|
||||
aggregation: 'latest-max'
|
||||
},
|
||||
{
|
||||
signal: 'cloud_sql.memory',
|
||||
type: 'cloudsql.googleapis.com/database/memory/utilization',
|
||||
resourceFilter: 'resource.type="cloudsql_database"',
|
||||
aggregation: 'latest-max'
|
||||
},
|
||||
{
|
||||
signal: 'cloud_sql.backends',
|
||||
type: 'cloudsql.googleapis.com/database/postgresql/num_backends',
|
||||
resourceFilter: 'resource.type="cloudsql_database"',
|
||||
aggregation: 'latest-sum'
|
||||
},
|
||||
{
|
||||
signal: 'cloud_sql.lock_waits',
|
||||
type: 'cloudsql.googleapis.com/database/postgresql/backends_in_wait',
|
||||
resourceFilter:
|
||||
'resource.type="cloudsql_database" AND metric.label."wait_event_type"="Lock"',
|
||||
aggregation: 'latest-max',
|
||||
emptyIsZero: true,
|
||||
zeroAfterMs: INCIDENT_MONITOR_THRESHOLDS.cloudLockWaitCarryMs
|
||||
},
|
||||
{
|
||||
signal: 'cloud_sql.deadlocks',
|
||||
type: 'cloudsql.googleapis.com/database/postgresql/deadlock_count',
|
||||
resourceFilter: 'resource.type="cloudsql_database"',
|
||||
aggregation: 'window-sum',
|
||||
emptyIsZero: true
|
||||
},
|
||||
{
|
||||
signal: 'director.instances',
|
||||
type: 'run.googleapis.com/container/instance_count',
|
||||
resourceFilter: 'resource.type="cloud_run_revision"',
|
||||
aggregation: 'latest-sum'
|
||||
},
|
||||
{
|
||||
signal: 'director.cpu',
|
||||
type: 'run.googleapis.com/container/cpu/utilizations',
|
||||
resourceFilter: 'resource.type="cloud_run_revision"',
|
||||
aggregation: 'latest-max'
|
||||
},
|
||||
{
|
||||
signal: 'director.memory',
|
||||
type: 'run.googleapis.com/container/memory/utilizations',
|
||||
resourceFilter: 'resource.type="cloud_run_revision"',
|
||||
aggregation: 'latest-max'
|
||||
},
|
||||
{
|
||||
signal: 'director.concurrency',
|
||||
type: 'run.googleapis.com/container/max_request_concurrencies',
|
||||
resourceFilter:
|
||||
'resource.type="cloud_run_revision" AND metric.label."state"="active"',
|
||||
aggregation: 'latest-max'
|
||||
},
|
||||
{
|
||||
signal: 'director.errors',
|
||||
type: 'run.googleapis.com/request_count',
|
||||
resourceFilter:
|
||||
'resource.type="cloud_run_revision" AND metric.label."response_code_class"="5xx" AND metric.label."response_code"!="503"',
|
||||
aggregation: 'window-sum',
|
||||
emptyIsZero: true
|
||||
},
|
||||
{
|
||||
signal: 'auth.errors',
|
||||
type: 'run.googleapis.com/request_count',
|
||||
resourceFilter:
|
||||
'resource.type="cloud_run_revision" AND metric.label."response_code_class"="5xx"',
|
||||
aggregation: 'window-sum',
|
||||
emptyIsZero: true
|
||||
},
|
||||
{
|
||||
signal: 'auth.requests',
|
||||
type: 'run.googleapis.com/request_count',
|
||||
resourceFilter: 'resource.type="cloud_run_revision"',
|
||||
aggregation: 'window-sum',
|
||||
emptyIsZero: true
|
||||
}
|
||||
]
|
||||
|
||||
export function pointValue(point: z.infer<typeof MonitoringPointSchema>): number {
|
||||
return (
|
||||
point.value.doubleValue ??
|
||||
point.value.int64Value ??
|
||||
point.value.distributionValue?.range?.max ??
|
||||
point.value.distributionValue?.mean ??
|
||||
0
|
||||
)
|
||||
}
|
||||
|
||||
function serviceFilter(signal: string, directorService: string, authService: string): string {
|
||||
if (signal.startsWith('director.')) {
|
||||
return `resource.label."service_name"="${directorService}"`
|
||||
}
|
||||
if (signal.startsWith('auth.')) {
|
||||
return `resource.label."service_name"="${authService}"`
|
||||
}
|
||||
return ''
|
||||
}
|
||||
|
||||
function targetFilter(
|
||||
definition: GoogleMetricDefinition,
|
||||
environment: RelayOpsEnvironment
|
||||
): string {
|
||||
if (definition.signal.startsWith('cloud_sql.')) {
|
||||
return `resource.label."database_id"="${environment.project}:${environment.sqlInstance}"`
|
||||
}
|
||||
return serviceFilter(
|
||||
definition.signal,
|
||||
environment.directorService,
|
||||
environment.authService
|
||||
)
|
||||
}
|
||||
|
||||
export async function googleJson(
|
||||
fetchImpl: typeof fetch,
|
||||
token: string,
|
||||
url: URL | string,
|
||||
init: RequestInit = {}
|
||||
): Promise<unknown> {
|
||||
const response = await fetchImpl(url, {
|
||||
...init,
|
||||
headers: {
|
||||
authorization: `Bearer ${token}`,
|
||||
...(init.body ? { 'content-type': 'application/json' } : {})
|
||||
},
|
||||
signal: AbortSignal.timeout(30_000)
|
||||
})
|
||||
if (!response.ok) throw new Error(`Google telemetry returned ${response.status}`)
|
||||
return await response.json()
|
||||
}
|
||||
|
||||
export async function readGoogleMetric(
|
||||
environment: RelayOpsEnvironment,
|
||||
definition: GoogleMetricDefinition,
|
||||
token: string,
|
||||
startAt: string,
|
||||
endAt: string,
|
||||
fetchImpl: typeof fetch,
|
||||
now: () => number = () => Date.parse(endAt)
|
||||
): Promise<IncidentSignal | null> {
|
||||
const url = new URL(
|
||||
`https://monitoring.googleapis.com/v3/projects/${environment.project}/timeSeries`
|
||||
)
|
||||
const filters = [
|
||||
`metric.type="${definition.type}"`,
|
||||
definition.resourceFilter,
|
||||
targetFilter(definition, environment)
|
||||
].filter(Boolean)
|
||||
url.searchParams.set('filter', filters.join(' AND '))
|
||||
url.searchParams.set('interval.startTime', startAt)
|
||||
url.searchParams.set('interval.endTime', endAt)
|
||||
url.searchParams.set('view', 'FULL')
|
||||
url.searchParams.set('pageSize', '1000')
|
||||
const parsed = MonitoringResponseSchema.parse(
|
||||
await googleJson(fetchImpl, token, url)
|
||||
)
|
||||
if (parsed.nextPageToken) throw new Error('Google metric pagination is incomplete')
|
||||
const queryEndMs = Date.parse(endAt)
|
||||
const readAtMs = Math.max(queryEndMs, now())
|
||||
const readAt = new Date(readAtMs).toISOString()
|
||||
const points = parsed.timeSeries.flatMap((series) => series.points)
|
||||
if (points.length === 0) {
|
||||
return definition.emptyIsZero ? { value: 0, observedAt: readAt } : null
|
||||
}
|
||||
const zeroAfterMs = definition.zeroAfterMs
|
||||
if (definition.emptyIsZero && zeroAfterMs !== undefined) {
|
||||
const latestSeriesPoints = parsed.timeSeries.flatMap((series) => {
|
||||
const seriesNewestAt = Math.max(
|
||||
...series.points.map((point) => Date.parse(point.interval.endTime))
|
||||
)
|
||||
return series.points.filter(
|
||||
(point) => Date.parse(point.interval.endTime) === seriesNewestAt
|
||||
)
|
||||
})
|
||||
const futurePoints = latestSeriesPoints.filter(
|
||||
(point) => Date.parse(point.interval.endTime) > queryEndMs
|
||||
)
|
||||
if (futurePoints.length > 0) {
|
||||
return {
|
||||
value: Math.max(...futurePoints.map(pointValue)),
|
||||
observedAt: new Date(Math.max(
|
||||
...futurePoints.map((point) => Date.parse(point.interval.endTime))
|
||||
)).toISOString()
|
||||
}
|
||||
}
|
||||
const recentNonzero = latestSeriesPoints.filter((point) => {
|
||||
const pointAt = Date.parse(point.interval.endTime)
|
||||
return pointValue(point) > 0 && queryEndMs - pointAt <= zeroAfterMs
|
||||
})
|
||||
if (recentNonzero.length === 0) return { value: 0, observedAt: readAt }
|
||||
return {
|
||||
value: Math.max(...recentNonzero.map(pointValue)),
|
||||
observedAt: new Date(Math.min(
|
||||
...recentNonzero.map((point) => Date.parse(point.interval.endTime))
|
||||
)).toISOString()
|
||||
}
|
||||
}
|
||||
const newestAt = Math.max(...points.map((point) => Date.parse(point.interval.endTime)))
|
||||
const selected = definition.aggregation === 'window-sum'
|
||||
? points
|
||||
: points.filter((point) => Date.parse(point.interval.endTime) === newestAt)
|
||||
const values = selected.map(pointValue)
|
||||
const value =
|
||||
definition.aggregation !== 'latest-max'
|
||||
? values.reduce((total, entry) => total + entry, 0)
|
||||
: Math.max(...values)
|
||||
return {
|
||||
value,
|
||||
observedAt: new Date(newestAt).toISOString()
|
||||
}
|
||||
}
|
||||
|
||||
export async function readGoogleMetricWithEmptyRetry(
|
||||
environment: RelayOpsEnvironment,
|
||||
definition: GoogleMetricDefinition,
|
||||
token: string,
|
||||
startAt: string,
|
||||
endAt: string,
|
||||
fetchImpl: typeof fetch,
|
||||
now: () => number = () => Date.parse(endAt),
|
||||
wait: (ms: number) => Promise<void> = async (ms) =>
|
||||
await new Promise((resolve) => setTimeout(resolve, ms))
|
||||
): Promise<IncidentSignal | null> {
|
||||
for (let attempt = 0; attempt < 3; attempt++) {
|
||||
const signal = await readGoogleMetric(
|
||||
environment,
|
||||
definition,
|
||||
token,
|
||||
startAt,
|
||||
endAt,
|
||||
fetchImpl,
|
||||
now
|
||||
)
|
||||
if (signal !== null || definition.emptyIsZero) return signal
|
||||
if (attempt < 2) await wait(2_000)
|
||||
}
|
||||
return null
|
||||
}
|
||||
|
||||
function addSignal(
|
||||
signals: Record<string, IncidentSignal>,
|
||||
name: string,
|
||||
value: number | null,
|
||||
observedAt: string | null
|
||||
): void {
|
||||
if (value === null || observedAt === null) return
|
||||
signals[name] = { value, observedAt }
|
||||
}
|
||||
|
||||
function endpointSignals(
|
||||
snapshot: Awaited<ReturnType<typeof buildDashboardSnapshot>>,
|
||||
nowAt: string
|
||||
): IncidentSource {
|
||||
const signals: Record<string, IncidentSignal> = {}
|
||||
const addEndpoint = (
|
||||
prefix: string,
|
||||
endpoint: { health: boolean | null; ready: boolean | null; latencyMs: number | null },
|
||||
unavailableIsZero = false
|
||||
) => {
|
||||
addSignal(
|
||||
signals,
|
||||
`${prefix}.health`,
|
||||
endpoint.health === null ? (unavailableIsZero ? 0 : null) : Number(endpoint.health),
|
||||
nowAt
|
||||
)
|
||||
addSignal(
|
||||
signals,
|
||||
`${prefix}.ready`,
|
||||
endpoint.ready === null ? (unavailableIsZero ? 0 : null) : Number(endpoint.ready),
|
||||
nowAt
|
||||
)
|
||||
addSignal(signals, `${prefix}.latency_ms`, endpoint.latencyMs, nowAt)
|
||||
}
|
||||
addEndpoint('director', snapshot.resources.directorEndpoint)
|
||||
addEndpoint('auth', snapshot.resources.authEndpoint)
|
||||
for (const cell of snapshot.resources.cells) {
|
||||
addEndpoint(`cell.${cell.cellId}`, cell.endpoint, true)
|
||||
}
|
||||
return { observedAt: nowAt, signals }
|
||||
}
|
||||
|
||||
export function relayFiveMinuteDeltaSignal(
|
||||
metric: Pick<RelayMetricSnapshot, 'available' | 'points'>,
|
||||
endAt: string
|
||||
): IncidentSignal | null {
|
||||
if (!metric.available) return null
|
||||
const endMs = Date.parse(endAt)
|
||||
if (!Number.isFinite(endMs)) throw new Error('Relay telemetry end time is invalid')
|
||||
return {
|
||||
value: metric.points
|
||||
.filter((point) => {
|
||||
const pointMs = Date.parse(point.at)
|
||||
return pointMs >= endMs - 300_000 && pointMs <= endMs
|
||||
})
|
||||
.reduce((total, point) => total + point.value, 0),
|
||||
observedAt: endAt
|
||||
}
|
||||
}
|
||||
|
||||
function relaySignals(
|
||||
snapshot: Awaited<ReturnType<typeof buildDashboardSnapshot>>
|
||||
): IncidentSource {
|
||||
const metrics = snapshot.monitoring.metrics
|
||||
const signals: Record<string, IncidentSignal> = {}
|
||||
addSignal(
|
||||
signals,
|
||||
'relay.pool_waiting',
|
||||
metrics.db_waiters_max.latest,
|
||||
metrics.db_waiters_max.latestAt
|
||||
)
|
||||
addSignal(
|
||||
signals,
|
||||
'relay.pool_wait_ms',
|
||||
metrics.db_wait_ms_max.latest,
|
||||
metrics.db_wait_ms_max.latestAt
|
||||
)
|
||||
const retries = relayFiveMinuteDeltaSignal(
|
||||
metrics.postgres_retries,
|
||||
snapshot.monitoring.endAt
|
||||
)
|
||||
if (retries) signals['relay.postgres_retries'] = retries
|
||||
const retryExhausted = relayFiveMinuteDeltaSignal(
|
||||
metrics.postgres_retry_exhausted,
|
||||
snapshot.monitoring.endAt
|
||||
)
|
||||
if (retryExhausted) signals['relay.postgres_retry_exhausted'] = retryExhausted
|
||||
for (const cell of snapshot.resources.cells) {
|
||||
addSignal(
|
||||
signals,
|
||||
`cell.${cell.cellId}.connections`,
|
||||
metrics.total_connections.latestByCell[cell.cellId] ?? null,
|
||||
metrics.total_connections.latestAt
|
||||
)
|
||||
addSignal(
|
||||
signals,
|
||||
`cell.${cell.cellId}.queued_bytes`,
|
||||
metrics.queued_bytes.latestByCell[cell.cellId] ?? null,
|
||||
metrics.queued_bytes.latestAt
|
||||
)
|
||||
}
|
||||
const observedTimes = Object.values(signals).map((entry) => Date.parse(entry.observedAt))
|
||||
if (observedTimes.length === 0) throw new Error('Relay telemetry is unavailable')
|
||||
const observedAt = new Date(Math.max(...observedTimes)).toISOString()
|
||||
return { observedAt, signals }
|
||||
}
|
||||
|
||||
const ADMIN_RETRY_ATTEMPTS = 3
|
||||
const ADMIN_RETRY_INTERVAL_MS = 5_000
|
||||
const ADMIN_RETRYABLE_STATUSES = new Set([500, 502, 503, 504])
|
||||
const AdminErrorBodySchema = z.object({ error: z.string() })
|
||||
// The director's /v1/admin/cell-status maps any thrown operation error onto 404
|
||||
// with an {error: message} body, so a Cloud SQL pool connect timeout arrives
|
||||
// here as a 404. These are the pool-acquire and transient SQLSTATE messages the
|
||||
// relay app's own isRelayDatabaseTransientError treats as re-runnable.
|
||||
const ADMIN_TRANSIENT_ERROR_MESSAGES = [
|
||||
'timeout exceeded when trying to connect',
|
||||
'Connection terminated due to connection timeout',
|
||||
'Connection terminated unexpectedly',
|
||||
'database_temporarily_unavailable',
|
||||
'deadlock detected',
|
||||
'canceling statement due to',
|
||||
'too many connections',
|
||||
'the database system is shutting down',
|
||||
'the database system is starting up'
|
||||
]
|
||||
|
||||
const defaultWait = async (ms: number): Promise<void> =>
|
||||
await new Promise((resolveWait) => setTimeout(resolveWait, ms))
|
||||
|
||||
// A dropped socket or the 30 s AbortSignal firing; both leave the request with
|
||||
// no verdict, and every admin read here is a plain query.
|
||||
function transientFetchFailure(error: unknown): boolean {
|
||||
if (error instanceof TypeError) return true
|
||||
const name = (error as { name?: unknown } | null)?.name
|
||||
return name === 'AbortError' || name === 'TimeoutError'
|
||||
}
|
||||
|
||||
function adminErrorMessage(body: string): string | null {
|
||||
const parsed = AdminErrorBodySchema.safeParse(
|
||||
((): unknown => {
|
||||
try {
|
||||
return JSON.parse(body)
|
||||
} catch {
|
||||
return null
|
||||
}
|
||||
})()
|
||||
)
|
||||
return parsed.success ? parsed.data.error : null
|
||||
}
|
||||
|
||||
function retryableAdminStatus(status: number, body: string): boolean {
|
||||
if (ADMIN_RETRYABLE_STATUSES.has(status)) return true
|
||||
// 401/403/400/413 and a 404 for 'director_only' are decisions, not weather.
|
||||
if (status !== 404) return false
|
||||
const message = adminErrorMessage(body)
|
||||
return (
|
||||
message !== null &&
|
||||
ADMIN_TRANSIENT_ERROR_MESSAGES.some((transient) => message.includes(transient))
|
||||
)
|
||||
}
|
||||
|
||||
// Step 5: cell-status refuses a reserve cell with exactly this 404 body; any other 404 stays a failure.
|
||||
class AdminReserveCellRefusal extends Error {
|
||||
constructor(status: number) {
|
||||
super(`Relay admin telemetry returned ${status}`)
|
||||
}
|
||||
}
|
||||
|
||||
async function adminPost(
|
||||
fetchImpl: typeof fetch,
|
||||
origin: string,
|
||||
token: string,
|
||||
path: string,
|
||||
body: unknown,
|
||||
wait: (ms: number) => Promise<void> = defaultWait
|
||||
): Promise<unknown> {
|
||||
for (let attempt = 1; ; attempt++) {
|
||||
const lastAttempt = attempt >= ADMIN_RETRY_ATTEMPTS
|
||||
let response: Response
|
||||
try {
|
||||
response = await fetchImpl(`${origin}${path}`, {
|
||||
method: 'POST',
|
||||
headers: { authorization: `Bearer ${token}`, 'content-type': 'application/json' },
|
||||
body: JSON.stringify(body),
|
||||
signal: AbortSignal.timeout(30_000)
|
||||
})
|
||||
} catch (error) {
|
||||
if (lastAttempt || !transientFetchFailure(error)) throw error
|
||||
console.warn(
|
||||
`relay admin telemetry retrying ${path} after network failure` +
|
||||
` (attempt ${attempt}/${ADMIN_RETRY_ATTEMPTS})`
|
||||
)
|
||||
await wait(ADMIN_RETRY_INTERVAL_MS)
|
||||
continue
|
||||
}
|
||||
if (response.ok) return await response.json()
|
||||
// Status only: the body carries a director error message, which must not
|
||||
// reach the log the way an identity would.
|
||||
const errorBody = await response.text().catch(() => '')
|
||||
if (response.status === 404 && adminErrorMessage(errorBody) === 'cell_in_reserve_mode') {
|
||||
throw new AdminReserveCellRefusal(response.status)
|
||||
}
|
||||
if (lastAttempt || !retryableAdminStatus(response.status, errorBody)) {
|
||||
throw new Error(`Relay admin telemetry returned ${response.status}`)
|
||||
}
|
||||
console.warn(
|
||||
`relay admin telemetry retrying ${path} after ${response.status}` +
|
||||
` (attempt ${attempt}/${ADMIN_RETRY_ATTEMPTS})`
|
||||
)
|
||||
await wait(ADMIN_RETRY_INTERVAL_MS)
|
||||
}
|
||||
}
|
||||
|
||||
export async function directorSignals(
|
||||
environmentId: RelayOpsEnvironmentId,
|
||||
expectedSelector: AdmissionSelector,
|
||||
gcloud: GcloudClient,
|
||||
nowMs: number,
|
||||
fetchImpl: typeof fetch,
|
||||
wait: (ms: number) => Promise<void> = defaultWait
|
||||
): Promise<{
|
||||
source: IncidentSource
|
||||
selector: AdmissionSelector
|
||||
cells: IncidentSample['cells']
|
||||
}> {
|
||||
const environment = relayOpsEnvironment(environmentId)
|
||||
if (!gcloud.identityToken) throw new Error('gcloud identity-token support is unavailable')
|
||||
const token = await gcloud.identityToken(`${environment.directorOrigin}/v1/admin/drain`)
|
||||
const configuredCellIds = new Set(environment.cells.map((cell) => cell.cellId))
|
||||
const rawSelector = SelectorStatusSchema.parse(
|
||||
await adminPost(
|
||||
fetchImpl,
|
||||
environment.directorOrigin,
|
||||
token,
|
||||
'/v1/admin/admission-selector/status',
|
||||
{ v: 1 },
|
||||
wait
|
||||
)
|
||||
).selector
|
||||
const selector = {
|
||||
generation: rawSelector.generation,
|
||||
membership: normalizeSelectorMembership(rawSelector.membership, configuredCellIds)
|
||||
}
|
||||
const statuses: Array<{
|
||||
cell: RelayOpsEnvironment['cells'][number]
|
||||
status: ({ reserve: false } & z.infer<typeof CellStatusSchema>['status']) | { reserve: true }
|
||||
}> = []
|
||||
for (const cell of environment.cells) {
|
||||
let response: unknown
|
||||
try {
|
||||
response = await adminPost(
|
||||
fetchImpl,
|
||||
environment.directorOrigin,
|
||||
token,
|
||||
'/v1/admin/cell-status',
|
||||
{ v: 1, cellId: cell.cellId },
|
||||
wait
|
||||
)
|
||||
} catch (error) {
|
||||
if (!(error instanceof AdminReserveCellRefusal)) throw error
|
||||
statuses.push({ cell, status: { reserve: true } })
|
||||
continue
|
||||
}
|
||||
statuses.push({
|
||||
cell,
|
||||
status: { reserve: false, ...CellStatusSchema.parse(response).status }
|
||||
})
|
||||
}
|
||||
const reserveCellIds = new Set(
|
||||
statuses.filter(({ status }) => status.reserve).map(({ cell }) => cell.cellId)
|
||||
)
|
||||
const migrationEntries: Array<{
|
||||
sourceCellId: string
|
||||
migration: z.infer<typeof MigrationStatusSchema>
|
||||
}> = []
|
||||
const migrationTargets = new Set(expectedSelector.membership.migrationOnly)
|
||||
for (const source of environment.cells) {
|
||||
for (const target of environment.cells) {
|
||||
if (source.cellId === target.cellId || !migrationTargets.has(target.cellId)) continue
|
||||
// The director refuses migration reads, and every migration, on a reserve cell.
|
||||
if (reserveCellIds.has(source.cellId)) continue
|
||||
migrationEntries.push({
|
||||
sourceCellId: source.cellId,
|
||||
migration: MigrationStatusSchema.parse(
|
||||
await adminPost(
|
||||
fetchImpl,
|
||||
environment.directorOrigin,
|
||||
token,
|
||||
'/v1/admin/evacuation-status',
|
||||
{
|
||||
v: 1,
|
||||
sourceCellId: source.cellId,
|
||||
targetCellId: target.cellId,
|
||||
completeReady: false
|
||||
},
|
||||
wait
|
||||
)
|
||||
)
|
||||
})
|
||||
}
|
||||
}
|
||||
const migrationBySource = new Map<string, { blocked: number; targetInactive: number }>()
|
||||
for (const { sourceCellId, migration } of migrationEntries) {
|
||||
const aggregate = migrationBySource.get(sourceCellId) ?? { blocked: 0, targetInactive: 0 }
|
||||
aggregate.blocked += migration.blocked + migration.blockedExpiredUnregistered
|
||||
aggregate.targetInactive += migration.registeredTargetInactive
|
||||
migrationBySource.set(sourceCellId, aggregate)
|
||||
}
|
||||
const nowAt = new Date(nowMs).toISOString()
|
||||
const signals: Record<string, IncidentSignal> = {}
|
||||
for (const { cell, status } of statuses) {
|
||||
const prefix = `cell.${cell.cellId}`
|
||||
if (status.reserve) {
|
||||
// cell-status carries the live cap; unset in Terraform keeps the evaluator's default, as there.
|
||||
if (cell.connectionHardCap !== null) {
|
||||
addSignal(signals, `${prefix}.connection_hard_cap`, cell.connectionHardCap, nowAt)
|
||||
}
|
||||
// Legacy admission needs relay_cells.enabled, which only cell-status reads: left missing.
|
||||
if (selector.generation !== 0) {
|
||||
addSignal(
|
||||
signals,
|
||||
`${prefix}.admission_state`,
|
||||
['existing-only', 'migration-only', 'general'].indexOf(
|
||||
selectorCellState(selector, cell.cellId)
|
||||
),
|
||||
nowAt
|
||||
)
|
||||
}
|
||||
continue
|
||||
}
|
||||
const admissionState = effectiveAdmissionState(
|
||||
selector,
|
||||
status.enabled,
|
||||
cell.cellId
|
||||
)
|
||||
addSignal(
|
||||
signals,
|
||||
`${prefix}.admission_state`,
|
||||
['existing-only', 'migration-only', 'general'].indexOf(admissionState),
|
||||
nowAt
|
||||
)
|
||||
addSignal(
|
||||
signals,
|
||||
`${prefix}.connection_hard_cap`,
|
||||
status.connectionCapacity?.hardCap ??
|
||||
INCIDENT_MONITOR_THRESHOLDS.cellConnections,
|
||||
nowAt
|
||||
)
|
||||
if (status.runtime) {
|
||||
addSignal(
|
||||
signals,
|
||||
`${prefix}.heartbeat_fresh`,
|
||||
Number(status.runtime.heartbeatFresh),
|
||||
nowAt
|
||||
)
|
||||
addSignal(
|
||||
signals,
|
||||
`${prefix}.heartbeat_age_ms`,
|
||||
Math.max(0, nowMs - status.runtime.lastHeartbeatAt),
|
||||
nowAt
|
||||
)
|
||||
}
|
||||
const migration = migrationBySource.get(cell.cellId) ?? { blocked: 0, targetInactive: 0 }
|
||||
addSignal(signals, `${prefix}.migration_blocked`, migration.blocked, nowAt)
|
||||
addSignal(
|
||||
signals,
|
||||
`${prefix}.migration_target_inactive`,
|
||||
migration.targetInactive,
|
||||
nowAt
|
||||
)
|
||||
}
|
||||
return {
|
||||
source: { observedAt: nowAt, signals },
|
||||
selector,
|
||||
cells: statuses.map(({ cell, status }) => ({
|
||||
cellId: cell.cellId,
|
||||
region: cell.region,
|
||||
runtimeKnown: true,
|
||||
powered: true,
|
||||
expectedAdmissionState: selectorCellState(expectedSelector, cell.cellId),
|
||||
...(status.reserve ? { reserve: true } : {})
|
||||
}))
|
||||
}
|
||||
}
|
||||
|
||||
export type IncidentSampleCollectorOptions = {
|
||||
environment: RelayOpsEnvironmentId
|
||||
expectedSelector: AdmissionSelector
|
||||
fetchImpl?: typeof fetch
|
||||
now?: () => number
|
||||
wait?: (ms: number) => Promise<void>
|
||||
}
|
||||
|
||||
export function createIncidentSampleCollector(
|
||||
gcloud: GcloudClient,
|
||||
options: IncidentSampleCollectorOptions
|
||||
): () => Promise<IncidentSample> {
|
||||
const fetchImpl = options.fetchImpl ?? fetch
|
||||
const now = options.now ?? Date.now
|
||||
return async () => {
|
||||
const nowMs = now()
|
||||
const nowAt = new Date(nowMs).toISOString()
|
||||
const startAt = new Date(nowMs - 5 * 60_000).toISOString()
|
||||
const environment = relayOpsEnvironment(options.environment)
|
||||
const accessToken = gcloud.accessToken()
|
||||
const cloudMetricEntries = accessToken.then(async (token) => await Promise.all(
|
||||
GOOGLE_METRICS.map(async (definition) => [
|
||||
definition.signal,
|
||||
await readGoogleMetricWithEmptyRetry(
|
||||
environment,
|
||||
definition,
|
||||
token,
|
||||
startAt,
|
||||
nowAt,
|
||||
fetchImpl,
|
||||
now
|
||||
)
|
||||
] as const)
|
||||
))
|
||||
const [snapshot, director, metricEntries] = await Promise.all([
|
||||
buildDashboardSnapshot(options.environment, gcloud, {
|
||||
windowMinutes: 5,
|
||||
now: new Date(nowMs),
|
||||
fetchImpl
|
||||
}),
|
||||
directorSignals(
|
||||
options.environment,
|
||||
options.expectedSelector,
|
||||
gcloud,
|
||||
nowMs,
|
||||
fetchImpl,
|
||||
options.wait ?? defaultWait
|
||||
),
|
||||
cloudMetricEntries
|
||||
])
|
||||
const cloudSignals = Object.fromEntries(
|
||||
metricEntries.filter((entry): entry is [string, IncidentSignal] => entry[1] !== null)
|
||||
)
|
||||
const relay = relaySignals(snapshot)
|
||||
const poweredByCell = new Map(
|
||||
snapshot.resources.cells.map((cell) => [
|
||||
cell.cellId,
|
||||
{
|
||||
runtimeKnown: cell.targetSize !== null,
|
||||
powered: (cell.targetSize ?? 0) > 0
|
||||
}
|
||||
])
|
||||
)
|
||||
return {
|
||||
collectedAt: nowAt,
|
||||
selector: director.selector,
|
||||
expectedSelector: options.expectedSelector,
|
||||
sources: {
|
||||
'active-probe': endpointSignals(snapshot, nowAt),
|
||||
'cloud-monitoring': { observedAt: nowAt, signals: cloudSignals },
|
||||
'relay-logs': relay,
|
||||
'director-admin': director.source
|
||||
},
|
||||
cells: director.cells.map((cell) => ({
|
||||
...cell,
|
||||
runtimeKnown: poweredByCell.get(cell.cellId)?.runtimeKnown ?? false,
|
||||
powered: poweredByCell.get(cell.cellId)?.powered ?? false
|
||||
}))
|
||||
}
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -1,77 +0,0 @@
|
||||
import { z } from 'zod'
|
||||
|
||||
export const AdmissionStateSchema = z.enum([
|
||||
'existing-only',
|
||||
'migration-only',
|
||||
'general'
|
||||
])
|
||||
|
||||
export type AdmissionState = z.infer<typeof AdmissionStateSchema>
|
||||
|
||||
export const SelectorMembershipSchema = z.object({
|
||||
existingOnly: z.array(z.string()),
|
||||
migrationOnly: z.array(z.string()),
|
||||
general: z.array(z.string())
|
||||
})
|
||||
|
||||
export type SelectorMembership = z.infer<typeof SelectorMembershipSchema>
|
||||
|
||||
export const AdmissionSelectorSchema = z.object({
|
||||
generation: z.number().int().nonnegative(),
|
||||
membership: SelectorMembershipSchema
|
||||
})
|
||||
|
||||
export type AdmissionSelector = z.infer<typeof AdmissionSelectorSchema>
|
||||
|
||||
export function normalizeSelectorMembership(
|
||||
membership: SelectorMembership,
|
||||
configuredCellIds: ReadonlySet<string>
|
||||
): SelectorMembership {
|
||||
const normalized = {
|
||||
existingOnly: [...membership.existingOnly].sort(),
|
||||
migrationOnly: [...membership.migrationOnly].sort(),
|
||||
general: [...membership.general].sort()
|
||||
}
|
||||
const all = [
|
||||
...normalized.existingOnly,
|
||||
...normalized.migrationOnly,
|
||||
...normalized.general
|
||||
]
|
||||
if (
|
||||
all.length !== configuredCellIds.size ||
|
||||
new Set(all).size !== all.length ||
|
||||
all.some((cellId) => !configuredCellIds.has(cellId))
|
||||
) {
|
||||
throw new Error('selector membership must contain every configured cell exactly once')
|
||||
}
|
||||
return normalized
|
||||
}
|
||||
|
||||
export function selectorCellState(
|
||||
selector: AdmissionSelector,
|
||||
cellId: string
|
||||
): AdmissionState {
|
||||
if (selector.membership.existingOnly.includes(cellId)) return 'existing-only'
|
||||
if (selector.membership.migrationOnly.includes(cellId)) return 'migration-only'
|
||||
if (selector.membership.general.includes(cellId)) return 'general'
|
||||
throw new Error(`selector does not contain ${cellId}`)
|
||||
}
|
||||
|
||||
export function effectiveAdmissionState(
|
||||
selector: AdmissionSelector,
|
||||
legacyEnabled: boolean,
|
||||
cellId: string
|
||||
): AdmissionState {
|
||||
if (selector.generation === 0) return legacyEnabled ? 'general' : 'existing-only'
|
||||
return selectorCellState(selector, cellId)
|
||||
}
|
||||
|
||||
export function exactAdmissionSelector(
|
||||
actual: AdmissionSelector,
|
||||
expected: AdmissionSelector
|
||||
): boolean {
|
||||
return (
|
||||
actual.generation === expected.generation &&
|
||||
JSON.stringify(actual.membership) === JSON.stringify(expected.membership)
|
||||
)
|
||||
}
|
||||
@@ -1,108 +0,0 @@
|
||||
import { randomBytes, timingSafeEqual } from 'node:crypto'
|
||||
import { readFile } from 'node:fs/promises'
|
||||
import { resolve } from 'node:path'
|
||||
import { serve } from '@hono/node-server'
|
||||
import { Hono } from 'hono'
|
||||
import { z } from 'zod'
|
||||
import { DashboardSnapshotCache } from './dashboard-snapshot.js'
|
||||
import { createGcloudClient } from './gcloud-client.js'
|
||||
import {
|
||||
dispatchStagingPowerWorkflow,
|
||||
parseStagingPowerRequest
|
||||
} from './staging-workflow.js'
|
||||
|
||||
const QuerySchema = z.object({
|
||||
environment: z.enum(['production', 'staging']).default('production'),
|
||||
window: z.coerce.number().int().min(30).max(1440).default(360)
|
||||
})
|
||||
const port = z.coerce.number().int().min(1024).max(65_535).parse(process.env.PORT ?? 2455)
|
||||
const controlsEnabled = process.env.RELAY_OPS_ENABLE_STAGING_CONTROLS === '1'
|
||||
const csrfToken = randomBytes(32).toString('base64url')
|
||||
const publicDirectory = resolve(import.meta.dirname, '../public')
|
||||
const cache = new DashboardSnapshotCache(createGcloudClient())
|
||||
const app = new Hono()
|
||||
|
||||
function safeEqual(left: string, right: string): boolean {
|
||||
const leftBuffer = Buffer.from(left)
|
||||
const rightBuffer = Buffer.from(right)
|
||||
return leftBuffer.length === rightBuffer.length && timingSafeEqual(leftBuffer, rightBuffer)
|
||||
}
|
||||
|
||||
app.use('*', async (context, next) => {
|
||||
await next()
|
||||
context.header('Cache-Control', 'no-store')
|
||||
context.header('Content-Security-Policy', [
|
||||
"default-src 'self'",
|
||||
"script-src 'self'",
|
||||
"style-src 'self'",
|
||||
"font-src 'self'",
|
||||
"connect-src 'self'",
|
||||
"img-src 'self' data:",
|
||||
"object-src 'none'",
|
||||
"base-uri 'none'",
|
||||
"frame-ancestors 'none'",
|
||||
"form-action 'self'"
|
||||
].join('; '))
|
||||
context.header('Referrer-Policy', 'no-referrer')
|
||||
context.header('X-Content-Type-Options', 'nosniff')
|
||||
context.header('X-Frame-Options', 'DENY')
|
||||
})
|
||||
|
||||
app.get('/health', (context) => context.json({ status: 'ok' }))
|
||||
app.get('/api/config', (context) => context.json({
|
||||
stagingControlsEnabled: controlsEnabled,
|
||||
csrfToken: controlsEnabled ? csrfToken : null
|
||||
}))
|
||||
app.get('/api/snapshot', async (context) => {
|
||||
const query = QuerySchema.safeParse(context.req.query())
|
||||
if (!query.success) return context.json({ error: 'Invalid dashboard query' }, 400)
|
||||
try {
|
||||
return context.json(await cache.read(query.data.environment, query.data.window))
|
||||
} catch {
|
||||
return context.json({
|
||||
error: 'Relay operations data is unavailable. Check local gcloud and gh authentication.'
|
||||
}, 503)
|
||||
}
|
||||
})
|
||||
app.post('/api/staging/power', async (context) => {
|
||||
if (!controlsEnabled) return context.json({ error: 'Staging controls are disabled' }, 403)
|
||||
const origin = context.req.header('origin')
|
||||
const expectedOrigin = `http://127.0.0.1:${port}`
|
||||
if (origin !== expectedOrigin) return context.json({ error: 'Origin rejected' }, 403)
|
||||
if (!safeEqual(context.req.header('x-csrf-token') ?? '', csrfToken)) {
|
||||
return context.json({ error: 'Request token rejected' }, 403)
|
||||
}
|
||||
try {
|
||||
const request = parseStagingPowerRequest(await context.req.json())
|
||||
await dispatchStagingPowerWorkflow(request)
|
||||
return context.json({ accepted: true })
|
||||
} catch {
|
||||
return context.json({ error: 'Invalid or failed staging workflow dispatch' }, 400)
|
||||
}
|
||||
})
|
||||
|
||||
const staticTypes: Record<string, string> = {
|
||||
'/app.js': 'text/javascript; charset=utf-8',
|
||||
'/styles.css': 'text/css; charset=utf-8'
|
||||
}
|
||||
for (const [route, contentType] of Object.entries(staticTypes)) {
|
||||
app.get(route, async (context) => {
|
||||
const file = route.slice(1)
|
||||
try {
|
||||
const content = await readFile(resolve(publicDirectory, file))
|
||||
return context.body(content, 200, { 'Content-Type': contentType })
|
||||
} catch {
|
||||
return context.notFound()
|
||||
}
|
||||
})
|
||||
}
|
||||
app.get('/', async (context) => {
|
||||
const html = await readFile(resolve(publicDirectory, 'index.html'), 'utf8')
|
||||
return context.html(html)
|
||||
})
|
||||
|
||||
serve({ fetch: app.fetch, hostname: '127.0.0.1', port }, () => {
|
||||
// Loopback is intentional; operators may add authenticated Tailscale Serve separately.
|
||||
console.log(`Orca Relay Operations: http://127.0.0.1:${port}`)
|
||||
console.log(`Staging controls: ${controlsEnabled ? 'enabled through GitHub workflow' : 'read-only'}`)
|
||||
})
|
||||
@@ -1,85 +0,0 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { RELAY_METRICS, readMonitoringSnapshot } from './monitoring-snapshot.js'
|
||||
import type { GcloudClient } from './gcloud-client.js'
|
||||
import { RELAY_OPS_ENVIRONMENTS } from './environment-config.js'
|
||||
|
||||
const gcloud: GcloudClient = {
|
||||
accessToken: async () => 'a'.repeat(40)
|
||||
}
|
||||
|
||||
function distribution(at: string, mean: number | undefined, count = 1) {
|
||||
return {
|
||||
interval: { endTime: at },
|
||||
value: { distributionValue: { count, ...(mean === undefined ? {} : { mean }) } }
|
||||
}
|
||||
}
|
||||
|
||||
describe('readMonitoringSnapshot', () => {
|
||||
it('aggregates gauge series per minute and distribution deltas by count', async () => {
|
||||
const fetchImpl: typeof fetch = async (input) => {
|
||||
const url = new URL(String(input))
|
||||
if (url.pathname.endsWith('/alertPolicies')) {
|
||||
return Response.json({ alertPolicies: [] })
|
||||
}
|
||||
const filter = url.searchParams.get('filter') ?? ''
|
||||
if (filter.includes('orca_relay_controls')) {
|
||||
return Response.json({ timeSeries: [
|
||||
{
|
||||
metric: { labels: { cell_id: 'production-gce-c1' } },
|
||||
resource: { type: 'gce_instance', labels: { instance_id: 'one' } },
|
||||
points: [
|
||||
distribution('2026-07-15T12:00:10Z', 1),
|
||||
distribution('2026-07-15T12:00:50Z', 2)
|
||||
]
|
||||
},
|
||||
{
|
||||
metric: { labels: { cell_id: 'production-gce-c1' } },
|
||||
resource: { type: 'gce_instance', labels: { instance_id: 'two' } },
|
||||
points: [distribution('2026-07-15T12:00:20Z', 3)]
|
||||
},
|
||||
{
|
||||
metric: { labels: { cell_id: 'production-gce-c1' } },
|
||||
resource: { type: 'gce_instance', labels: { instance_id: 'stale' } },
|
||||
points: [distribution('2026-07-15T11:59:20Z', 100)]
|
||||
}
|
||||
] })
|
||||
}
|
||||
if (filter.includes('orca_relay_forwarded_bytes')) {
|
||||
return Response.json({ timeSeries: [{
|
||||
metric: { labels: { cell_id: 'production-gce-c1' } },
|
||||
resource: { type: 'gce_instance', labels: { instance_id: 'one' } },
|
||||
points: [distribution('2026-07-15T12:00:20Z', 10, 4)]
|
||||
}] })
|
||||
}
|
||||
return Response.json({ timeSeries: [] })
|
||||
}
|
||||
|
||||
const result = await readMonitoringSnapshot(RELAY_OPS_ENVIRONMENTS.production, gcloud, {
|
||||
now: new Date('2026-07-15T12:01:00Z'),
|
||||
windowMinutes: 30,
|
||||
fetchImpl
|
||||
})
|
||||
|
||||
expect(result.warnings).toEqual([])
|
||||
expect(result.metrics.controls.points).toEqual([
|
||||
{ at: '2026-07-15T11:59:00.000Z', value: 100 },
|
||||
{ at: '2026-07-15T12:00:00.000Z', value: 5 }
|
||||
])
|
||||
expect(result.metrics.controls.latestByCell).toEqual({ 'production-gce-c1': 5 })
|
||||
expect(result.metrics.forwarded_bytes.latest).toBe(40)
|
||||
expect(result.metrics.postgres_retries.available).toBe(true)
|
||||
expect(Object.keys(result.metrics)).toHaveLength(RELAY_METRICS.length)
|
||||
})
|
||||
|
||||
it('degrades safely when credentials are unavailable', async () => {
|
||||
const unavailable: GcloudClient = {
|
||||
accessToken: async () => { throw new Error('sensitive context') }
|
||||
}
|
||||
const result = await readMonitoringSnapshot(RELAY_OPS_ENVIRONMENTS.production, unavailable)
|
||||
expect(result.warnings).toEqual([
|
||||
'Cloud Monitoring credentials are unavailable. Run gcloud auth login.'
|
||||
])
|
||||
expect(result.metrics.postgres_retries.available).toBe(false)
|
||||
expect(JSON.stringify(result)).not.toContain('sensitive context')
|
||||
})
|
||||
})
|
||||
@@ -1,379 +0,0 @@
|
||||
import { z } from 'zod'
|
||||
import type { RelayOpsEnvironment } from './environment-config.js'
|
||||
import type { GcloudClient } from './gcloud-client.js'
|
||||
|
||||
type MetricMode = 'gauge-sum' | 'delta-sum' | 'maximum'
|
||||
|
||||
export type RelayMetricName =
|
||||
| 'total_connections'
|
||||
| 'controls'
|
||||
| 'splices'
|
||||
| 'pending_splices'
|
||||
| 'queued_bytes'
|
||||
| 'http_latency_ms'
|
||||
| 'sql_latency_ms'
|
||||
| 'heap_used_bytes'
|
||||
| 'event_loop_ms_p99'
|
||||
| 'forwarded_bytes'
|
||||
| 'auth_successes'
|
||||
| 'auth_failures'
|
||||
| 'reconnects'
|
||||
| 'sql_queries'
|
||||
| 'sql_failures'
|
||||
| 'assignment_5xx'
|
||||
| 'postgres_retries'
|
||||
| 'postgres_retry_exhausted'
|
||||
| 'db_pool_total'
|
||||
| 'db_pool_idle'
|
||||
| 'db_pool_waiting'
|
||||
| 'db_waiters_max'
|
||||
| 'db_oldest_wait_ms'
|
||||
| 'db_wait_ms_max'
|
||||
|
||||
type MetricDefinition = {
|
||||
name: RelayMetricName
|
||||
label: string
|
||||
unit: 'count' | 'bytes' | 'milliseconds'
|
||||
mode: MetricMode
|
||||
}
|
||||
|
||||
export const RELAY_METRICS: MetricDefinition[] = [
|
||||
{ name: 'total_connections', label: 'Connections', unit: 'count', mode: 'gauge-sum' },
|
||||
{ name: 'controls', label: 'Desktop controls', unit: 'count', mode: 'gauge-sum' },
|
||||
{ name: 'splices', label: 'Phone splices', unit: 'count', mode: 'gauge-sum' },
|
||||
{ name: 'pending_splices', label: 'Pending splices', unit: 'count', mode: 'gauge-sum' },
|
||||
{ name: 'queued_bytes', label: 'Queued bytes', unit: 'bytes', mode: 'maximum' },
|
||||
{ name: 'http_latency_ms', label: 'HTTP latency', unit: 'milliseconds', mode: 'maximum' },
|
||||
{ name: 'sql_latency_ms', label: 'SQL latency', unit: 'milliseconds', mode: 'maximum' },
|
||||
{ name: 'heap_used_bytes', label: 'Heap used', unit: 'bytes', mode: 'maximum' },
|
||||
{
|
||||
name: 'event_loop_ms_p99',
|
||||
label: 'Event-loop p99',
|
||||
unit: 'milliseconds',
|
||||
mode: 'maximum'
|
||||
},
|
||||
{ name: 'forwarded_bytes', label: 'Forwarded bytes', unit: 'bytes', mode: 'delta-sum' },
|
||||
{ name: 'auth_successes', label: 'Auth successes', unit: 'count', mode: 'delta-sum' },
|
||||
{ name: 'auth_failures', label: 'Auth failures', unit: 'count', mode: 'delta-sum' },
|
||||
{ name: 'reconnects', label: 'Reconnects', unit: 'count', mode: 'delta-sum' },
|
||||
{ name: 'sql_queries', label: 'SQL queries', unit: 'count', mode: 'delta-sum' },
|
||||
{ name: 'sql_failures', label: 'SQL failures', unit: 'count', mode: 'delta-sum' },
|
||||
{ name: 'assignment_5xx', label: 'Assignment 5xx', unit: 'count', mode: 'delta-sum' },
|
||||
{
|
||||
name: 'postgres_retries',
|
||||
label: 'PostgreSQL retries',
|
||||
unit: 'count',
|
||||
mode: 'delta-sum'
|
||||
},
|
||||
{
|
||||
name: 'postgres_retry_exhausted',
|
||||
label: 'PostgreSQL retry exhausted',
|
||||
unit: 'count',
|
||||
mode: 'delta-sum'
|
||||
},
|
||||
{ name: 'db_pool_total', label: 'Database pool total', unit: 'count', mode: 'gauge-sum' },
|
||||
{ name: 'db_pool_idle', label: 'Database pool idle', unit: 'count', mode: 'gauge-sum' },
|
||||
{
|
||||
name: 'db_pool_waiting',
|
||||
label: 'Database pool waiting',
|
||||
unit: 'count',
|
||||
mode: 'gauge-sum'
|
||||
},
|
||||
{
|
||||
name: 'db_waiters_max',
|
||||
label: 'Database waiters max',
|
||||
unit: 'count',
|
||||
mode: 'maximum'
|
||||
},
|
||||
{
|
||||
name: 'db_oldest_wait_ms',
|
||||
label: 'Database oldest wait',
|
||||
unit: 'milliseconds',
|
||||
mode: 'maximum'
|
||||
},
|
||||
{
|
||||
name: 'db_wait_ms_max',
|
||||
label: 'Database wait max',
|
||||
unit: 'milliseconds',
|
||||
mode: 'maximum'
|
||||
}
|
||||
]
|
||||
|
||||
const NumericSchema = z.union([z.number(), z.string()]).transform((value) => Number(value))
|
||||
const DistributionSchema = z.object({
|
||||
count: NumericSchema.default(0),
|
||||
mean: NumericSchema.optional()
|
||||
})
|
||||
const PointSchema = z.object({
|
||||
interval: z.object({ endTime: z.string() }),
|
||||
value: z.object({
|
||||
doubleValue: NumericSchema.optional(),
|
||||
int64Value: NumericSchema.optional(),
|
||||
distributionValue: DistributionSchema.optional()
|
||||
})
|
||||
})
|
||||
const TimeSeriesSchema = z.object({
|
||||
metric: z.object({ labels: z.record(z.string(), z.string()).default({}) }),
|
||||
resource: z.object({ type: z.string(), labels: z.record(z.string(), z.string()).default({}) }),
|
||||
points: z.array(PointSchema).default([])
|
||||
})
|
||||
const TimeSeriesResponseSchema = z.object({
|
||||
timeSeries: z.array(TimeSeriesSchema).default([])
|
||||
})
|
||||
|
||||
export type MetricPoint = { at: string; value: number }
|
||||
|
||||
export type RelayMetricSnapshot = MetricDefinition & {
|
||||
available: boolean
|
||||
points: MetricPoint[]
|
||||
latest: number | null
|
||||
latestAt: string | null
|
||||
latestByCell: Record<string, number>
|
||||
}
|
||||
|
||||
export type AlertPolicySnapshot = {
|
||||
id: string
|
||||
displayName: string
|
||||
enabled: boolean
|
||||
documentation: string | null
|
||||
}
|
||||
|
||||
export type MonitoringSnapshot = {
|
||||
startAt: string
|
||||
endAt: string
|
||||
resolutionSeconds: number
|
||||
metrics: Record<RelayMetricName, RelayMetricSnapshot>
|
||||
alertPolicies: AlertPolicySnapshot[]
|
||||
warnings: string[]
|
||||
}
|
||||
|
||||
type ParsedPoint = {
|
||||
atMs: number
|
||||
bucketMs: number
|
||||
value: number
|
||||
sampleTotal: number
|
||||
seriesKey: string
|
||||
cellId: string
|
||||
}
|
||||
|
||||
function pointValue(point: z.infer<typeof PointSchema>): { value: number; sampleTotal: number } {
|
||||
if (point.value.distributionValue) {
|
||||
const count = point.value.distributionValue.count
|
||||
const value = point.value.distributionValue.mean ?? 0
|
||||
return { value, sampleTotal: value * count }
|
||||
}
|
||||
const value = point.value.doubleValue ?? point.value.int64Value ?? 0
|
||||
return { value, sampleTotal: value }
|
||||
}
|
||||
|
||||
function parsePoints(series: z.infer<typeof TimeSeriesSchema>[]): ParsedPoint[] {
|
||||
return series.flatMap((entry) => {
|
||||
const cellId = entry.metric.labels.cell_id ?? 'unknown'
|
||||
const resourceId =
|
||||
entry.resource.labels.instance_id ?? entry.resource.labels.revision_name ?? entry.resource.type
|
||||
const seriesKey = `${cellId}:${resourceId}`
|
||||
return entry.points.flatMap((point) => {
|
||||
const atMs = Date.parse(point.interval.endTime)
|
||||
if (!Number.isFinite(atMs)) return []
|
||||
const values = pointValue(point)
|
||||
return [{
|
||||
atMs,
|
||||
bucketMs: Math.floor(atMs / 60_000) * 60_000,
|
||||
value: values.value,
|
||||
sampleTotal: values.sampleTotal,
|
||||
seriesKey,
|
||||
cellId
|
||||
}]
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
function aggregatePoints(points: ParsedPoint[], mode: MetricMode): MetricPoint[] {
|
||||
if (mode === 'gauge-sum') {
|
||||
const buckets = new Map<number, Map<string, ParsedPoint>>()
|
||||
for (const point of points) {
|
||||
const bySeries = buckets.get(point.bucketMs) ?? new Map<string, ParsedPoint>()
|
||||
const previous = bySeries.get(point.seriesKey)
|
||||
if (!previous || previous.atMs < point.atMs) bySeries.set(point.seriesKey, point)
|
||||
buckets.set(point.bucketMs, bySeries)
|
||||
}
|
||||
return [...buckets.entries()]
|
||||
.sort(([left], [right]) => left - right)
|
||||
.map(([at, bySeries]) => ({
|
||||
at: new Date(at).toISOString(),
|
||||
value: [...bySeries.values()].reduce((total, point) => total + point.value, 0)
|
||||
}))
|
||||
}
|
||||
const buckets = new Map<number, number>()
|
||||
for (const point of points) {
|
||||
const value = mode === 'delta-sum' ? point.sampleTotal : point.value
|
||||
const previous = buckets.get(point.bucketMs)
|
||||
buckets.set(
|
||||
point.bucketMs,
|
||||
mode === 'maximum' ? Math.max(previous ?? 0, value) : (previous ?? 0) + value
|
||||
)
|
||||
}
|
||||
return [...buckets.entries()]
|
||||
.sort(([left], [right]) => left - right)
|
||||
.map(([at, value]) => ({ at: new Date(at).toISOString(), value }))
|
||||
}
|
||||
|
||||
function latestByCell(points: ParsedPoint[], mode: MetricMode): Record<string, number> {
|
||||
const newestBucketByCell = new Map<string, number>()
|
||||
for (const point of points) {
|
||||
newestBucketByCell.set(
|
||||
point.cellId,
|
||||
Math.max(newestBucketByCell.get(point.cellId) ?? 0, point.bucketMs)
|
||||
)
|
||||
}
|
||||
const newestBySeries = new Map<string, ParsedPoint>()
|
||||
for (const point of points) {
|
||||
if (point.bucketMs !== newestBucketByCell.get(point.cellId)) continue
|
||||
const previous = newestBySeries.get(point.seriesKey)
|
||||
if (!previous || previous.atMs < point.atMs) newestBySeries.set(point.seriesKey, point)
|
||||
}
|
||||
const totals = new Map<string, number>()
|
||||
for (const point of newestBySeries.values()) {
|
||||
const value = mode === 'delta-sum' ? point.sampleTotal : point.value
|
||||
const previous = totals.get(point.cellId)
|
||||
totals.set(
|
||||
point.cellId,
|
||||
mode === 'maximum' ? Math.max(previous ?? 0, value) : (previous ?? 0) + value
|
||||
)
|
||||
}
|
||||
return Object.fromEntries(totals)
|
||||
}
|
||||
|
||||
async function monitoringRequest(
|
||||
fetchImpl: typeof fetch,
|
||||
token: string,
|
||||
url: URL
|
||||
): Promise<unknown> {
|
||||
const response = await fetchImpl(url, {
|
||||
headers: { authorization: `Bearer ${token}` },
|
||||
signal: AbortSignal.timeout(30_000)
|
||||
})
|
||||
if (!response.ok) throw new Error(`Cloud Monitoring returned ${response.status}`)
|
||||
return await response.json()
|
||||
}
|
||||
|
||||
async function readMetric(
|
||||
environment: RelayOpsEnvironment,
|
||||
definition: MetricDefinition,
|
||||
token: string,
|
||||
startAt: string,
|
||||
endAt: string,
|
||||
fetchImpl: typeof fetch
|
||||
): Promise<RelayMetricSnapshot> {
|
||||
const url = new URL(
|
||||
`https://monitoring.googleapis.com/v3/projects/${environment.project}/timeSeries`
|
||||
)
|
||||
url.searchParams.set(
|
||||
'filter',
|
||||
`metric.type="logging.googleapis.com/user/orca_relay_${definition.name}"`
|
||||
)
|
||||
url.searchParams.set('interval.startTime', startAt)
|
||||
url.searchParams.set('interval.endTime', endAt)
|
||||
url.searchParams.set('view', 'FULL')
|
||||
url.searchParams.set('pageSize', '1000')
|
||||
const body = TimeSeriesResponseSchema.parse(
|
||||
await monitoringRequest(fetchImpl, token, url)
|
||||
)
|
||||
const parsed = parsePoints(body.timeSeries)
|
||||
const points = aggregatePoints(parsed, definition.mode)
|
||||
const latest = points.at(-1) ?? null
|
||||
return {
|
||||
...definition,
|
||||
available: true,
|
||||
points,
|
||||
latest: latest?.value ?? null,
|
||||
latestAt: latest?.at ?? null,
|
||||
latestByCell: latestByCell(parsed, definition.mode)
|
||||
}
|
||||
}
|
||||
|
||||
async function readAlertPolicies(
|
||||
environment: RelayOpsEnvironment,
|
||||
token: string,
|
||||
fetchImpl: typeof fetch
|
||||
): Promise<AlertPolicySnapshot[]> {
|
||||
const url = new URL(
|
||||
`https://monitoring.googleapis.com/v3/projects/${environment.project}/alertPolicies`
|
||||
)
|
||||
url.searchParams.set('pageSize', '100')
|
||||
const body = (await monitoringRequest(fetchImpl, token, url)) as {
|
||||
alertPolicies?: Array<{
|
||||
name?: string
|
||||
displayName?: string
|
||||
enabled?: boolean
|
||||
documentation?: { content?: string }
|
||||
}>
|
||||
}
|
||||
return (body.alertPolicies ?? [])
|
||||
.filter((policy) => policy.displayName?.startsWith('Orca Relay:'))
|
||||
.map((policy) => ({
|
||||
id: policy.name ?? '',
|
||||
displayName: policy.displayName ?? 'Orca Relay alert',
|
||||
enabled: policy.enabled === true,
|
||||
documentation: policy.documentation?.content ?? null
|
||||
}))
|
||||
.sort((left, right) => left.displayName.localeCompare(right.displayName))
|
||||
}
|
||||
|
||||
function emptyMetric(definition: MetricDefinition): RelayMetricSnapshot {
|
||||
return {
|
||||
...definition,
|
||||
available: false,
|
||||
points: [],
|
||||
latest: null,
|
||||
latestAt: null,
|
||||
latestByCell: {}
|
||||
}
|
||||
}
|
||||
|
||||
export async function readMonitoringSnapshot(
|
||||
environment: RelayOpsEnvironment,
|
||||
gcloud: GcloudClient,
|
||||
options: { now?: Date; windowMinutes?: number; fetchImpl?: typeof fetch } = {}
|
||||
): Promise<MonitoringSnapshot> {
|
||||
const now = options.now ?? new Date()
|
||||
const windowMinutes = Math.min(24 * 60, Math.max(30, options.windowMinutes ?? 360))
|
||||
const endAt = now.toISOString()
|
||||
const startAt = new Date(now.getTime() - windowMinutes * 60_000).toISOString()
|
||||
const fetchImpl = options.fetchImpl ?? fetch
|
||||
const warnings: string[] = []
|
||||
let token: string
|
||||
try {
|
||||
token = await gcloud.accessToken()
|
||||
} catch {
|
||||
return {
|
||||
startAt,
|
||||
endAt,
|
||||
resolutionSeconds: 60,
|
||||
metrics: Object.fromEntries(
|
||||
RELAY_METRICS.map((definition) => [definition.name, emptyMetric(definition)])
|
||||
) as Record<RelayMetricName, RelayMetricSnapshot>,
|
||||
alertPolicies: [],
|
||||
warnings: ['Cloud Monitoring credentials are unavailable. Run gcloud auth login.']
|
||||
}
|
||||
}
|
||||
const settled = await Promise.allSettled(
|
||||
RELAY_METRICS.map((definition) =>
|
||||
readMetric(environment, definition, token, startAt, endAt, fetchImpl)
|
||||
)
|
||||
)
|
||||
const metrics = {} as Record<RelayMetricName, RelayMetricSnapshot>
|
||||
settled.forEach((result, index) => {
|
||||
const definition = RELAY_METRICS[index]!
|
||||
if (result.status === 'fulfilled') metrics[definition.name] = result.value
|
||||
else {
|
||||
metrics[definition.name] = emptyMetric(definition)
|
||||
warnings.push(`${definition.label} metric is unavailable.`)
|
||||
}
|
||||
})
|
||||
const alertPolicies = await readAlertPolicies(environment, token, fetchImpl).catch(() => {
|
||||
warnings.push('Cloud Monitoring alert policies are unavailable.')
|
||||
return []
|
||||
})
|
||||
return { startAt, endAt, resolutionSeconds: 60, metrics, alertPolicies, warnings }
|
||||
}
|
||||
@@ -1,210 +0,0 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { relayOpsEnvironment } from './environment-config.js'
|
||||
import {
|
||||
CELL_PROCESS_EXIT_METRIC,
|
||||
createPreDrainHardRuleReader,
|
||||
INSTANCE_CELL_LOOKBACK_MS,
|
||||
type PreDrainExitScope
|
||||
} from './pre-drain-hard-rule-sources.js'
|
||||
|
||||
const environment = relayOpsEnvironment('production')
|
||||
const now = Date.parse('2026-10-01T12:00:00.000Z')
|
||||
const scope: PreDrainExitScope = {
|
||||
targetCellId: 'production-gce-c25',
|
||||
placementCellIds: new Set(['production-gce-c25', 'production-gce-c26']),
|
||||
configuredCellIds: new Set(environment.cells.map((cell) => cell.cellId))
|
||||
}
|
||||
|
||||
function points(values: number[], kind: 'int64Value' | 'doubleValue' = 'int64Value') {
|
||||
return values.map((value, index) => ({
|
||||
interval: { endTime: new Date(now - index * 60_000).toISOString() },
|
||||
value: { [kind]: kind === 'int64Value' ? String(value) : value }
|
||||
}))
|
||||
}
|
||||
|
||||
type Fake = {
|
||||
// instance id -> per-minute exit counts
|
||||
exits?: Record<string, number[]>
|
||||
// instance id -> the cell its runtime-metrics lines name; absent means it logged none
|
||||
cells?: Record<string, string>
|
||||
director503?: number[][]
|
||||
concurrency?: number[][]
|
||||
descriptorStatus?: number
|
||||
loggingStatus?: number
|
||||
}
|
||||
|
||||
function fakeGoogle(fake: Fake) {
|
||||
const requests: { url: URL; body: string | null }[] = []
|
||||
const fetchImpl: typeof fetch = async (input, init) => {
|
||||
const url = new URL(String(input))
|
||||
const body = typeof init?.body === 'string' ? init.body : null
|
||||
requests.push({ url, body })
|
||||
if (url.pathname.includes('/metricDescriptors/')) {
|
||||
return new Response('{}', { status: fake.descriptorStatus ?? 200 })
|
||||
}
|
||||
if (url.hostname === 'logging.googleapis.com') {
|
||||
if (fake.loggingStatus) return new Response('{}', { status: fake.loggingStatus })
|
||||
const filter = String(JSON.parse(body ?? '{}').filter ?? '')
|
||||
const instanceId = /instance_id="([0-9]+)"/.exec(filter)?.[1] ?? ''
|
||||
const cellId = fake.cells?.[instanceId]
|
||||
return Response.json({ entries: cellId ? [{ jsonPayload: { cellId } }] : [] })
|
||||
}
|
||||
const filter = url.searchParams.get('filter') ?? ''
|
||||
if (filter.includes(CELL_PROCESS_EXIT_METRIC)) {
|
||||
return Response.json({
|
||||
timeSeries: Object.entries(fake.exits ?? {}).map(([instanceId, values]) => ({
|
||||
resource: { labels: { instance_id: instanceId } },
|
||||
points: points(values)
|
||||
}))
|
||||
})
|
||||
}
|
||||
const series = filter.includes('request_count')
|
||||
? (fake.director503 ?? []).map((values) => points(values))
|
||||
: (fake.concurrency ?? [[1]]).map((values) => points(values, 'doubleValue'))
|
||||
return Response.json({ timeSeries: series.map((entry) => ({ points: entry })) })
|
||||
}
|
||||
return { fetchImpl, requests }
|
||||
}
|
||||
|
||||
function reader(fetchImpl: typeof fetch) {
|
||||
return createPreDrainHardRuleReader(environment, async () => 'token', scope, {
|
||||
fetchImpl,
|
||||
now: () => now
|
||||
})
|
||||
}
|
||||
|
||||
describe('pre-drain hard rule reader', () => {
|
||||
it('takes the busiest 503 minute and the highest concurrency p99', async () => {
|
||||
const { fetchImpl } = fakeGoogle({
|
||||
director503: [[40, 620, 55]],
|
||||
concurrency: [[12.5, 66.8]]
|
||||
})
|
||||
await expect(reader(fetchImpl)()).resolves.toEqual({
|
||||
cellProcessExits: 0,
|
||||
unattributedExitInstances: [],
|
||||
director503PeakPerMinute: 620,
|
||||
directorConcurrencyP99: 66.8
|
||||
})
|
||||
})
|
||||
|
||||
it('reads an empty concurrency series as unknown, not calm', async () => {
|
||||
const { fetchImpl } = fakeGoogle({ concurrency: [] })
|
||||
await expect(reader(fetchImpl)()).resolves.toMatchObject({ directorConcurrencyP99: null })
|
||||
})
|
||||
|
||||
it('ignores the target cell rolling the fix for its own crashes', async () => {
|
||||
const { fetchImpl } = fakeGoogle({
|
||||
exits: { '2525': [3, 4] },
|
||||
cells: { '2525': 'production-gce-c25' }
|
||||
})
|
||||
await expect(reader(fetchImpl)()).resolves.toMatchObject({
|
||||
cellProcessExits: 0,
|
||||
unattributedExitInstances: []
|
||||
})
|
||||
})
|
||||
|
||||
it('counts an exit on another cell that takes placements', async () => {
|
||||
const { fetchImpl } = fakeGoogle({
|
||||
exits: { '2626': [1], '2525': [5] },
|
||||
cells: { '2626': 'production-gce-c26', '2525': 'production-gce-c25' }
|
||||
})
|
||||
await expect(reader(fetchImpl)()).resolves.toMatchObject({
|
||||
cellProcessExits: 1,
|
||||
unattributedExitInstances: []
|
||||
})
|
||||
})
|
||||
|
||||
it('ignores an existing-only legacy cell', async () => {
|
||||
const { fetchImpl } = fakeGoogle({
|
||||
exits: { '505': [15] },
|
||||
cells: { '505': 'production-gce-c5' }
|
||||
})
|
||||
await expect(reader(fetchImpl)()).resolves.toMatchObject({
|
||||
cellProcessExits: 0,
|
||||
unattributedExitInstances: []
|
||||
})
|
||||
})
|
||||
|
||||
it('reports an exit it cannot name a cell for as unattributed', async () => {
|
||||
const { fetchImpl } = fakeGoogle({ exits: { '999': [1] } })
|
||||
await expect(reader(fetchImpl)()).resolves.toMatchObject({
|
||||
cellProcessExits: 0,
|
||||
unattributedExitInstances: ['999']
|
||||
})
|
||||
})
|
||||
|
||||
it('fails the read, rather than guessing, when the cell lookup itself fails', async () => {
|
||||
const { fetchImpl } = fakeGoogle({ exits: { '2626': [1] }, loggingStatus: 403 })
|
||||
await expect(reader(fetchImpl)()).rejects.toThrow('Google telemetry returned 403')
|
||||
})
|
||||
|
||||
it('names an instance from its own runtime-metrics lines, once', async () => {
|
||||
const { fetchImpl, requests } = fakeGoogle({
|
||||
exits: { '2626': [1] },
|
||||
cells: { '2626': 'production-gce-c26' }
|
||||
})
|
||||
const read = reader(fetchImpl)
|
||||
await read()
|
||||
await read()
|
||||
const lookups = requests.filter(({ url }) => url.hostname === 'logging.googleapis.com')
|
||||
expect(lookups).toHaveLength(1)
|
||||
const body = JSON.parse(lookups[0]?.body ?? '{}')
|
||||
expect(body.filter).toContain('resource.labels.instance_id="2626"')
|
||||
expect(body.filter).toContain('jsonPayload.event="orca_relay_runtime_metrics"')
|
||||
expect(body.filter).toContain(
|
||||
`timestamp>="${new Date(now - INSTANCE_CELL_LOOKBACK_MS).toISOString()}"`
|
||||
)
|
||||
expect(body.orderBy).toBe('timestamp desc')
|
||||
})
|
||||
|
||||
it('asks again next sample for an instance it could not name', async () => {
|
||||
const fake: Fake = { exits: { '2626': [1] } }
|
||||
const { fetchImpl, requests } = fakeGoogle(fake)
|
||||
const read = reader(fetchImpl)
|
||||
await expect(read()).resolves.toMatchObject({ unattributedExitInstances: ['2626'] })
|
||||
fake.cells = { '2626': 'production-gce-c26' }
|
||||
await expect(read()).resolves.toMatchObject({
|
||||
cellProcessExits: 1,
|
||||
unattributedExitInstances: []
|
||||
})
|
||||
expect(requests.filter(({ url }) => url.hostname === 'logging.googleapis.com'))
|
||||
.toHaveLength(2)
|
||||
})
|
||||
|
||||
it('aligns each read per minute over its own lookback on the director service', async () => {
|
||||
const { fetchImpl, requests } = fakeGoogle({})
|
||||
await reader(fetchImpl)()
|
||||
const reads = requests.map(({ url }) => url).filter((url) => url.pathname.endsWith('/timeSeries'))
|
||||
expect(reads).toHaveLength(3)
|
||||
const byMetric = (needle: string): URL => {
|
||||
const url = reads.find((entry) => entry.searchParams.get('filter')?.includes(needle))
|
||||
if (!url) throw new Error(`no read for ${needle}`)
|
||||
return url
|
||||
}
|
||||
const exits = byMetric(CELL_PROCESS_EXIT_METRIC)
|
||||
expect(exits.searchParams.get('interval.startTime')).toBe('2026-10-01T11:50:00.000Z')
|
||||
expect(exits.searchParams.get('aggregation.groupByFields'))
|
||||
.toBe('resource.label."instance_id"')
|
||||
const director503 = byMetric('run.googleapis.com/request_count')
|
||||
expect(director503.searchParams.get('filter')).toContain('metric.label."response_code"="503"')
|
||||
expect(director503.searchParams.get('filter')).toContain(
|
||||
`resource.label."service_name"="${environment.directorService}"`
|
||||
)
|
||||
expect(director503.searchParams.get('aggregation.perSeriesAligner')).toBe('ALIGN_DELTA')
|
||||
expect(director503.searchParams.get('aggregation.crossSeriesReducer')).toBe('REDUCE_SUM')
|
||||
const concurrency = byMetric('max_request_concurrencies')
|
||||
expect(concurrency.searchParams.get('interval.startTime')).toBe('2026-10-01T11:56:00.000Z')
|
||||
expect(concurrency.searchParams.get('aggregation.perSeriesAligner'))
|
||||
.toBe('ALIGN_PERCENTILE_99')
|
||||
for (const url of reads) {
|
||||
expect(url.searchParams.get('aggregation.alignmentPeriod')).toBe('60s')
|
||||
expect(url.searchParams.get('interval.endTime')).toBe('2026-10-01T12:00:00.000Z')
|
||||
}
|
||||
})
|
||||
|
||||
// Why: a deleted or renamed log metric reads as an empty series, which is zero exits forever.
|
||||
it('refuses to count exits from a metric that does not exist', async () => {
|
||||
const { fetchImpl } = fakeGoogle({ descriptorStatus: 404 })
|
||||
await expect(reader(fetchImpl)()).rejects.toThrow('Google telemetry returned 404')
|
||||
})
|
||||
})
|
||||
@@ -1,217 +0,0 @@
|
||||
import { z } from 'zod'
|
||||
import type { RelayOpsEnvironment } from './environment-config.js'
|
||||
import {
|
||||
googleJson,
|
||||
MonitoringPointSchema,
|
||||
MonitoringResponseSchema,
|
||||
pointValue
|
||||
} from './incident-monitor-sources.js'
|
||||
import {
|
||||
attributeCellExits,
|
||||
PRE_DRAIN_HARD_RULES,
|
||||
type PreDrainHardRuleReadings
|
||||
} from './pre-drain-sample.js'
|
||||
|
||||
// The Docker `container die` count per relay cell, from infra/terraform/relay-observability.tf.
|
||||
export const CELL_PROCESS_EXIT_METRIC = 'logging.googleapis.com/user/orca_relay_cell_process_exit'
|
||||
|
||||
// How far back a cell's own runtime-metrics line is searched to name an instance. The instance id
|
||||
// survives a container restart, so a crash-looping container is still named by its earlier lines.
|
||||
export const INSTANCE_CELL_LOOKBACK_MS = 2 * 60 * 60_000
|
||||
|
||||
const ExitsByInstanceSchema = z.object({
|
||||
timeSeries: z.array(z.object({
|
||||
resource: z.object({ labels: z.object({ instance_id: z.string().regex(/^[0-9]+$/) }) }),
|
||||
points: z.array(MonitoringPointSchema)
|
||||
})).default([]),
|
||||
nextPageToken: z.string().optional()
|
||||
})
|
||||
|
||||
const RuntimeMetricsEntriesSchema = z.object({
|
||||
entries: z.array(z.object({
|
||||
jsonPayload: z.object({ cellId: z.string() })
|
||||
})).default([])
|
||||
})
|
||||
|
||||
export type PreDrainExitScope = {
|
||||
targetCellId: string
|
||||
placementCellIds: ReadonlySet<string>
|
||||
configuredCellIds: ReadonlySet<string>
|
||||
}
|
||||
|
||||
type AlignedRead = {
|
||||
filter: string
|
||||
lookbackMs: number
|
||||
aligner: 'ALIGN_DELTA' | 'ALIGN_PERCENTILE_99'
|
||||
reducer: 'REDUCE_SUM' | 'REDUCE_MAX'
|
||||
}
|
||||
|
||||
// Server-side per-minute alignment, so a burst is counted rather than sampled and no read can be
|
||||
// truncated the way a log read is.
|
||||
async function readPerMinute(
|
||||
environment: RelayOpsEnvironment,
|
||||
token: string,
|
||||
fetchImpl: typeof fetch,
|
||||
nowMs: number,
|
||||
read: AlignedRead
|
||||
): Promise<number[]> {
|
||||
const url = new URL(
|
||||
`https://monitoring.googleapis.com/v3/projects/${environment.project}/timeSeries`
|
||||
)
|
||||
url.searchParams.set('filter', read.filter)
|
||||
url.searchParams.set('interval.startTime', new Date(nowMs - read.lookbackMs).toISOString())
|
||||
url.searchParams.set('interval.endTime', new Date(nowMs).toISOString())
|
||||
url.searchParams.set('aggregation.alignmentPeriod', '60s')
|
||||
url.searchParams.set('aggregation.perSeriesAligner', read.aligner)
|
||||
url.searchParams.set('aggregation.crossSeriesReducer', read.reducer)
|
||||
url.searchParams.set('pageSize', '1000')
|
||||
const parsed = MonitoringResponseSchema.parse(await googleJson(fetchImpl, token, url))
|
||||
if (parsed.nextPageToken) throw new Error('Google metric pagination is incomplete')
|
||||
return parsed.timeSeries.flatMap((series) => series.points.map(pointValue))
|
||||
}
|
||||
|
||||
function directorFilter(environment: RelayOpsEnvironment, metricType: string): string {
|
||||
return [
|
||||
`metric.type="${metricType}"`,
|
||||
'resource.type="cloud_run_revision"',
|
||||
`resource.label."service_name"="${environment.directorService}"`
|
||||
].join(' AND ')
|
||||
}
|
||||
|
||||
async function readExitsByInstance(
|
||||
environment: RelayOpsEnvironment,
|
||||
token: string,
|
||||
fetchImpl: typeof fetch,
|
||||
nowMs: number
|
||||
): Promise<Map<string, number>> {
|
||||
const url = new URL(
|
||||
`https://monitoring.googleapis.com/v3/projects/${environment.project}/timeSeries`
|
||||
)
|
||||
url.searchParams.set(
|
||||
'filter',
|
||||
`metric.type="${CELL_PROCESS_EXIT_METRIC}" AND resource.type="gce_instance"`
|
||||
)
|
||||
url.searchParams.set(
|
||||
'interval.startTime',
|
||||
new Date(nowMs - PRE_DRAIN_HARD_RULES.cellProcessExitLookbackMs).toISOString()
|
||||
)
|
||||
url.searchParams.set('interval.endTime', new Date(nowMs).toISOString())
|
||||
url.searchParams.set('aggregation.alignmentPeriod', '60s')
|
||||
url.searchParams.set('aggregation.perSeriesAligner', 'ALIGN_SUM')
|
||||
url.searchParams.set('aggregation.crossSeriesReducer', 'REDUCE_SUM')
|
||||
url.searchParams.set('aggregation.groupByFields', 'resource.label."instance_id"')
|
||||
url.searchParams.set('pageSize', '1000')
|
||||
const parsed = ExitsByInstanceSchema.parse(await googleJson(fetchImpl, token, url))
|
||||
if (parsed.nextPageToken) throw new Error('Google metric pagination is incomplete')
|
||||
const exits = new Map<string, number>()
|
||||
for (const series of parsed.timeSeries) {
|
||||
const instanceId = series.resource.labels.instance_id
|
||||
const total = series.points.map(pointValue).reduce((sum, value) => sum + value, 0)
|
||||
exits.set(instanceId, (exits.get(instanceId) ?? 0) + total)
|
||||
}
|
||||
return exits
|
||||
}
|
||||
|
||||
// The cell an instance serves, from that instance's own newest runtime-metrics line; the exit
|
||||
// metric carries only the instance id. Null when the instance logged none in the lookback.
|
||||
async function readInstanceCell(
|
||||
environment: RelayOpsEnvironment,
|
||||
token: string,
|
||||
fetchImpl: typeof fetch,
|
||||
nowMs: number,
|
||||
instanceId: string
|
||||
): Promise<string | null> {
|
||||
const since = new Date(nowMs - INSTANCE_CELL_LOOKBACK_MS).toISOString()
|
||||
const body = RuntimeMetricsEntriesSchema.parse(await googleJson(
|
||||
fetchImpl,
|
||||
token,
|
||||
'https://logging.googleapis.com/v2/entries:list',
|
||||
{
|
||||
method: 'POST',
|
||||
body: JSON.stringify({
|
||||
resourceNames: [`projects/${environment.project}`],
|
||||
filter: [
|
||||
'resource.type="gce_instance"',
|
||||
`resource.labels.instance_id="${instanceId}"`,
|
||||
'jsonPayload.event="orca_relay_runtime_metrics"',
|
||||
`timestamp>="${since}"`
|
||||
].join(' AND '),
|
||||
orderBy: 'timestamp desc',
|
||||
pageSize: 1
|
||||
})
|
||||
}
|
||||
))
|
||||
return body.entries[0]?.jsonPayload.cellId ?? null
|
||||
}
|
||||
|
||||
export function createPreDrainHardRuleReader(
|
||||
environment: RelayOpsEnvironment,
|
||||
accessToken: () => Promise<string>,
|
||||
scope: PreDrainExitScope,
|
||||
options: { fetchImpl?: typeof fetch; now?: () => number } = {}
|
||||
): () => Promise<PreDrainHardRuleReadings> {
|
||||
const fetchImpl = options.fetchImpl ?? fetch
|
||||
const now = options.now ?? Date.now
|
||||
// An exit count of zero is only evidence if the metric exists; a renamed or deleted log metric
|
||||
// reads as an empty, calm series forever. Checked once, before the first reading is trusted.
|
||||
let descriptorChecked: Promise<void> | null = null
|
||||
const requireExitMetric = (token: string): Promise<void> => {
|
||||
descriptorChecked ??= googleJson(
|
||||
fetchImpl,
|
||||
token,
|
||||
`https://monitoring.googleapis.com/v3/projects/${environment.project}` +
|
||||
`/metricDescriptors/${CELL_PROCESS_EXIT_METRIC}`
|
||||
).then(() => undefined, (error: unknown) => {
|
||||
descriptorChecked = null
|
||||
throw error
|
||||
})
|
||||
return descriptorChecked
|
||||
}
|
||||
// An instance belongs to one cell for its whole life, so a resolved name is kept; an unresolved
|
||||
// one is asked again next sample.
|
||||
const cellByInstance = new Map<string, string>()
|
||||
return async () => {
|
||||
const token = await accessToken()
|
||||
await requireExitMetric(token)
|
||||
const nowMs = now()
|
||||
const [exitsByInstance, director503, concurrency] = await Promise.all([
|
||||
readExitsByInstance(environment, token, fetchImpl, nowMs),
|
||||
readPerMinute(environment, token, fetchImpl, nowMs, {
|
||||
filter: `${directorFilter(environment, 'run.googleapis.com/request_count')}` +
|
||||
' AND metric.label."response_code"="503"',
|
||||
lookbackMs: PRE_DRAIN_HARD_RULES.director503LookbackMs,
|
||||
aligner: 'ALIGN_DELTA',
|
||||
reducer: 'REDUCE_SUM'
|
||||
}),
|
||||
readPerMinute(environment, token, fetchImpl, nowMs, {
|
||||
filter: `${directorFilter(
|
||||
environment,
|
||||
'run.googleapis.com/container/max_request_concurrencies'
|
||||
)} AND metric.label."state"="active"`,
|
||||
lookbackMs: PRE_DRAIN_HARD_RULES.directorConcurrencyLookbackMs,
|
||||
aligner: 'ALIGN_PERCENTILE_99',
|
||||
reducer: 'REDUCE_MAX'
|
||||
})
|
||||
])
|
||||
const resolved = new Map<string, string | null>()
|
||||
for (const [instanceId, exits] of exitsByInstance) {
|
||||
if (exits <= 0) continue
|
||||
const known = cellByInstance.get(instanceId)
|
||||
const cellId = known ?? await readInstanceCell(environment, token, fetchImpl, nowMs, instanceId)
|
||||
if (cellId !== null) cellByInstance.set(instanceId, cellId)
|
||||
resolved.set(instanceId, cellId)
|
||||
}
|
||||
const attribution = attributeCellExits({
|
||||
exitsByInstance,
|
||||
cellByInstance: resolved,
|
||||
...scope
|
||||
})
|
||||
return {
|
||||
cellProcessExits: attribution.counted,
|
||||
unattributedExitInstances: attribution.unattributed,
|
||||
// 503s are a counter, so a minute with no point had none.
|
||||
director503PeakPerMinute: Math.max(0, ...director503),
|
||||
directorConcurrencyP99: concurrency.length === 0 ? null : Math.max(...concurrency)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,101 +0,0 @@
|
||||
import { mkdtempSync, rmSync, writeFileSync } from 'node:fs'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { join } from 'node:path'
|
||||
import { afterEach, describe, expect, it } from 'vitest'
|
||||
import { relayOpsEnvironment } from './environment-config.js'
|
||||
import { parsePreDrainSampleArgs, runPreDrainSampleCli } from './pre-drain-sample-cli.js'
|
||||
|
||||
const directories: string[] = []
|
||||
|
||||
afterEach(() => {
|
||||
for (const directory of directories.splice(0)) {
|
||||
rmSync(directory, { recursive: true, force: true })
|
||||
}
|
||||
})
|
||||
|
||||
function membershipFile(): string {
|
||||
const directory = mkdtempSync(join(tmpdir(), 'relay-pre-drain-selector-'))
|
||||
directories.push(directory)
|
||||
const path = join(directory, 'selector.json')
|
||||
writeFileSync(path, JSON.stringify({
|
||||
existingOnly: relayOpsEnvironment('production').cells.map((cell) => cell.cellId),
|
||||
migrationOnly: [],
|
||||
general: []
|
||||
}))
|
||||
return path
|
||||
}
|
||||
|
||||
function args(overrides: Record<string, string> = {}): string[] {
|
||||
const values: Record<string, string> = {
|
||||
'--target-cell-id': 'production-gce-c25',
|
||||
'--target-hosts': '1200',
|
||||
'--expected-selector-generation': '40',
|
||||
'--selector-membership-file': membershipFile(),
|
||||
'--wave-index': '2',
|
||||
'--selector-wave-delta': '2',
|
||||
...overrides
|
||||
}
|
||||
return Object.entries(values).flatMap(([name, value]) => [name, value])
|
||||
}
|
||||
|
||||
describe('pre-drain sample CLI', () => {
|
||||
it('requires every option exactly once and nothing else', () => {
|
||||
expect(() => parsePreDrainSampleArgs(['--', ...args()])).not.toThrow()
|
||||
const full = args()
|
||||
for (let index = 0; index < full.length; index += 2) {
|
||||
const missing = [...full.slice(0, index), ...full.slice(index + 2)]
|
||||
expect(() => parsePreDrainSampleArgs(missing)).toThrow('usage')
|
||||
}
|
||||
expect(() => parsePreDrainSampleArgs([...full, '--wave-index', '2'])).toThrow('usage')
|
||||
expect(() => parsePreDrainSampleArgs([...full, '--skip-window', '1'])).toThrow('usage')
|
||||
expect(() => parsePreDrainSampleArgs(args({ '--target-hosts': '-1' }))).toThrow('usage')
|
||||
expect(() => parsePreDrainSampleArgs(args({ '--target-hosts': '' }))).toThrow('usage')
|
||||
})
|
||||
|
||||
it('rejects a target cell the environment does not configure', async () => {
|
||||
await expect(runPreDrainSampleCli(args({ '--target-cell-id': 'production-gce-c99' }), {
|
||||
collect: async () => { throw new Error('must not sample') },
|
||||
readHardRules: async () => { throw new Error('must not sample') }
|
||||
})).rejects.toThrow('target cell is unknown')
|
||||
})
|
||||
|
||||
it('rejects a wave index or selector delta no wave produces', async () => {
|
||||
for (const overrides of [{ '--wave-index': '10' }, { '--selector-wave-delta': '1' }]) {
|
||||
await expect(runPreDrainSampleCli(args(overrides), {
|
||||
collect: async () => { throw new Error('must not sample') },
|
||||
readHardRules: async () => { throw new Error('must not sample') }
|
||||
})).rejects.toThrow('wave index or selector wave delta is invalid')
|
||||
}
|
||||
})
|
||||
|
||||
it('sizes the window from the target hosts and reports the verdict', async () => {
|
||||
let clock = Date.parse('2026-10-01T12:00:00.000Z')
|
||||
const lines: string[] = []
|
||||
let collected = 0
|
||||
// A sample that cannot be judged green trips the run, which is enough to see the window.
|
||||
await expect(runPreDrainSampleCli(args(), {
|
||||
now: () => clock,
|
||||
wait: async (ms) => { clock += ms },
|
||||
collect: async () => {
|
||||
collected += 1
|
||||
throw new Error('collector down')
|
||||
},
|
||||
readHardRules: async () => ({
|
||||
cellProcessExits: 0,
|
||||
unattributedExitInstances: [],
|
||||
director503PeakPerMinute: 0,
|
||||
directorConcurrencyP99: 1
|
||||
}),
|
||||
print: (line) => lines.push(line)
|
||||
})).rejects.toThrow('relay pre-drain sample tripped')
|
||||
const events = lines.map((line) => JSON.parse(line))
|
||||
expect(events[0]).toEqual({
|
||||
event: 'relay_pre_drain_sample_window',
|
||||
targetCellId: 'production-gce-c25',
|
||||
targetHosts: 1200,
|
||||
windowMinutes: 5
|
||||
})
|
||||
expect(events.at(-1)).toMatchObject({ event: 'relay_pre_drain_sample_tripped' })
|
||||
expect(collected).toBe(3)
|
||||
})
|
||||
})
|
||||
@@ -1,158 +0,0 @@
|
||||
import { pathToFileURL } from 'node:url'
|
||||
import { relayOpsEnvironment } from './environment-config.js'
|
||||
import { createGcloudClient } from './gcloud-client.js'
|
||||
import {
|
||||
livePreflightGcloud,
|
||||
readDispatchSelector,
|
||||
waveAdjustedSelector
|
||||
} from './incident-live-preflight-cli.js'
|
||||
import { createIncidentSampleCollector } from './incident-monitor-sources.js'
|
||||
import type { IncidentSample } from './incident-monitor.js'
|
||||
import { createPreDrainHardRuleReader } from './pre-drain-hard-rule-sources.js'
|
||||
import {
|
||||
preDrainSampleWindowMinutes,
|
||||
PreDrainSampleTripped,
|
||||
runPreDrainSample,
|
||||
type PreDrainHardRuleReadings
|
||||
} from './pre-drain-sample.js'
|
||||
|
||||
const USAGE =
|
||||
'usage: --target-cell-id <cell> --target-hosts <n> --expected-selector-generation <n>' +
|
||||
' --selector-membership-file <json> --wave-index <0-9> --selector-wave-delta <0|2>'
|
||||
|
||||
const OPTIONS = [
|
||||
'--target-cell-id',
|
||||
'--target-hosts',
|
||||
'--expected-selector-generation',
|
||||
'--selector-membership-file',
|
||||
'--wave-index',
|
||||
'--selector-wave-delta'
|
||||
] as const
|
||||
|
||||
type Option = (typeof OPTIONS)[number]
|
||||
|
||||
const COUNT = /^(0|[1-9][0-9]*)$/
|
||||
|
||||
function isOption(value: string): value is Option {
|
||||
return (OPTIONS as readonly string[]).includes(value)
|
||||
}
|
||||
|
||||
// Every option is required exactly once, so a typo can never silently shorten the window.
|
||||
export function parsePreDrainSampleArgs(argv: string[]): Record<Option, string> {
|
||||
const args = argv[0] === '--' ? argv.slice(1) : argv
|
||||
const values = new Map<Option, string>()
|
||||
for (let index = 0; index < args.length; index += 2) {
|
||||
const name = args[index] ?? ''
|
||||
const value = args[index + 1]
|
||||
if (!isOption(name) || values.has(name) || !value) throw new Error(USAGE)
|
||||
values.set(name, value)
|
||||
}
|
||||
const get = (name: Option): string => {
|
||||
const value = values.get(name)
|
||||
if (value === undefined) throw new Error(USAGE)
|
||||
return value
|
||||
}
|
||||
const parsed = {
|
||||
'--target-cell-id': get('--target-cell-id'),
|
||||
'--target-hosts': get('--target-hosts'),
|
||||
'--expected-selector-generation': get('--expected-selector-generation'),
|
||||
'--selector-membership-file': get('--selector-membership-file'),
|
||||
'--wave-index': get('--wave-index'),
|
||||
'--selector-wave-delta': get('--selector-wave-delta')
|
||||
}
|
||||
if (
|
||||
!COUNT.test(parsed['--target-hosts']) ||
|
||||
!COUNT.test(parsed['--expected-selector-generation'])
|
||||
) throw new Error(USAGE)
|
||||
return parsed
|
||||
}
|
||||
|
||||
export async function runPreDrainSampleCli(
|
||||
argv: string[],
|
||||
dependencies: {
|
||||
now?: () => number
|
||||
wait?: (ms: number) => Promise<void>
|
||||
collect?: () => Promise<IncidentSample>
|
||||
readHardRules?: () => Promise<PreDrainHardRuleReadings>
|
||||
gcloud?: ReturnType<typeof createGcloudClient>
|
||||
environment?: NodeJS.ProcessEnv
|
||||
print?: (line: string) => void
|
||||
} = {}
|
||||
): Promise<void> {
|
||||
const options = parsePreDrainSampleArgs(argv)
|
||||
const environment = relayOpsEnvironment('production')
|
||||
const configuredCellIds = new Set(environment.cells.map((cell) => cell.cellId))
|
||||
const targetCellId = options['--target-cell-id']
|
||||
// An unknown target would exempt nothing and make every one of its exits unattributed.
|
||||
if (!configuredCellIds.has(targetCellId)) throw new Error('pre-drain sample target cell is unknown')
|
||||
const windowMinutes = preDrainSampleWindowMinutes(Number(options['--target-hosts']))
|
||||
const expectedSelector = waveAdjustedSelector(
|
||||
await readDispatchSelector(
|
||||
options['--expected-selector-generation'],
|
||||
options['--selector-membership-file']
|
||||
),
|
||||
options['--wave-index'],
|
||||
options['--selector-wave-delta']
|
||||
)
|
||||
const print = dependencies.print ?? ((line: string) => console.log(line))
|
||||
const gcloud = livePreflightGcloud(
|
||||
dependencies.gcloud ?? createGcloudClient(),
|
||||
dependencies.environment
|
||||
)
|
||||
const collect = dependencies.collect ?? createIncidentSampleCollector(gcloud, {
|
||||
environment: 'production',
|
||||
expectedSelector,
|
||||
...(dependencies.now ? { now: dependencies.now } : {})
|
||||
})
|
||||
const { membership } = expectedSelector
|
||||
const readHardRules = dependencies.readHardRules ?? createPreDrainHardRuleReader(
|
||||
environment,
|
||||
() => gcloud.accessToken(),
|
||||
{
|
||||
targetCellId,
|
||||
placementCellIds: new Set([...membership.general, ...membership.migrationOnly]),
|
||||
configuredCellIds
|
||||
},
|
||||
dependencies.now ? { now: dependencies.now } : {}
|
||||
)
|
||||
print(JSON.stringify({
|
||||
event: 'relay_pre_drain_sample_window',
|
||||
targetCellId,
|
||||
targetHosts: Number(options['--target-hosts']),
|
||||
windowMinutes
|
||||
}))
|
||||
try {
|
||||
const result = await runPreDrainSample({
|
||||
windowMinutes,
|
||||
collect,
|
||||
readHardRules,
|
||||
...(dependencies.now ? { now: dependencies.now } : {}),
|
||||
...(dependencies.wait ? { wait: dependencies.wait } : {}),
|
||||
log: (record) => print(JSON.stringify({ event: 'relay_pre_drain_sample', ...record }))
|
||||
})
|
||||
print(JSON.stringify({
|
||||
event: 'relay_pre_drain_sample_passed',
|
||||
windowMinutes: result.windowMinutes,
|
||||
startedAt: result.startedAt,
|
||||
completedAt: result.completedAt,
|
||||
samples: result.samples.length
|
||||
}))
|
||||
} catch (error) {
|
||||
if (error instanceof PreDrainSampleTripped) {
|
||||
print(JSON.stringify({
|
||||
event: 'relay_pre_drain_sample_tripped',
|
||||
windowMinutes,
|
||||
samples: error.samples.length,
|
||||
failures: error.failures
|
||||
}))
|
||||
}
|
||||
throw error
|
||||
}
|
||||
}
|
||||
|
||||
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
|
||||
runPreDrainSampleCli(process.argv.slice(2)).catch((error: unknown) => {
|
||||
console.error(error instanceof Error ? error.message : 'relay pre-drain sample failed')
|
||||
process.exitCode = 1
|
||||
})
|
||||
}
|
||||
@@ -1,358 +0,0 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import {
|
||||
INCIDENT_FRESHNESS_TOLERANCE_SAMPLES,
|
||||
INCIDENT_MONITOR_THRESHOLDS,
|
||||
type IncidentSample
|
||||
} from './incident-monitor.js'
|
||||
import {
|
||||
attributeCellExits,
|
||||
evaluatePreDrainHardRules,
|
||||
PRE_DRAIN_HARD_RULES,
|
||||
PRE_DRAIN_MAX_OVERRUN_SAMPLES,
|
||||
preDrainSampleWindowMinutes,
|
||||
PreDrainSampleTripped,
|
||||
runPreDrainSample,
|
||||
type PreDrainHardRuleReadings
|
||||
} from './pre-drain-sample.js'
|
||||
|
||||
const selector = {
|
||||
generation: 1,
|
||||
membership: { existingOnly: ['production-gce-c1'], migrationOnly: [], general: [] }
|
||||
}
|
||||
|
||||
function greenSample(at: number): IncidentSample {
|
||||
const observedAt = new Date(at).toISOString()
|
||||
const signal = (value: number) => ({ value, observedAt })
|
||||
const source = (signals: Record<string, number>) => ({
|
||||
observedAt,
|
||||
signals: Object.fromEntries(
|
||||
Object.entries(signals).map(([name, value]) => [name, signal(value)])
|
||||
)
|
||||
})
|
||||
return {
|
||||
collectedAt: observedAt,
|
||||
selector,
|
||||
expectedSelector: selector,
|
||||
cells: [{
|
||||
cellId: 'production-gce-c1',
|
||||
region: 'us-central1',
|
||||
runtimeKnown: true,
|
||||
powered: true,
|
||||
expectedAdmissionState: 'existing-only'
|
||||
}],
|
||||
sources: {
|
||||
'active-probe': source({
|
||||
'director.health': 1,
|
||||
'director.ready': 1,
|
||||
'director.latency_ms': 1,
|
||||
'auth.health': 1,
|
||||
'auth.ready': 1,
|
||||
'auth.latency_ms': 1,
|
||||
'cell.production-gce-c1.health': 1,
|
||||
'cell.production-gce-c1.ready': 1,
|
||||
'cell.production-gce-c1.latency_ms': 1
|
||||
}),
|
||||
'cloud-monitoring': source({
|
||||
'cloud_sql.cpu': 0.1,
|
||||
'cloud_sql.memory': 0.1,
|
||||
'cloud_sql.backends': 1,
|
||||
'cloud_sql.lock_waits': 0,
|
||||
'cloud_sql.deadlocks': 0,
|
||||
'director.instances': 5,
|
||||
'director.cpu': 0.1,
|
||||
'director.memory': 0.1,
|
||||
'director.concurrency': 1,
|
||||
'director.errors': 0,
|
||||
'auth.errors': 0
|
||||
}),
|
||||
'relay-logs': source({
|
||||
'relay.pool_waiting': 0,
|
||||
'relay.pool_wait_ms': 0,
|
||||
'relay.postgres_retries': 0,
|
||||
'relay.postgres_retry_exhausted': 0,
|
||||
'cell.production-gce-c1.connections': 1,
|
||||
'cell.production-gce-c1.queued_bytes': 0
|
||||
}),
|
||||
'director-admin': source({
|
||||
'cell.production-gce-c1.admission_state': 0,
|
||||
'cell.production-gce-c1.heartbeat_fresh': 1,
|
||||
'cell.production-gce-c1.heartbeat_age_ms': 1,
|
||||
'cell.production-gce-c1.migration_blocked': 0,
|
||||
'cell.production-gce-c1.migration_target_inactive': 0
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const calm: PreDrainHardRuleReadings = {
|
||||
cellProcessExits: 0,
|
||||
unattributedExitInstances: [],
|
||||
director503PeakPerMinute: 40,
|
||||
directorConcurrencyP99: 20
|
||||
}
|
||||
|
||||
// A fake clock the sampler's own waits advance, so a window runs in no real time.
|
||||
function harness(input: {
|
||||
windowMinutes: number
|
||||
sampleAt?: (index: number, at: number) => IncidentSample | Error
|
||||
hardRulesAt?: (index: number) => PreDrainHardRuleReadings | Error
|
||||
}) {
|
||||
let clock = Date.parse('2026-10-01T12:00:00.000Z')
|
||||
let samples = 0
|
||||
let reads = 0
|
||||
const run = runPreDrainSample({
|
||||
windowMinutes: input.windowMinutes,
|
||||
now: () => clock,
|
||||
wait: async (ms) => {
|
||||
clock += ms
|
||||
},
|
||||
collect: async () => {
|
||||
const result = input.sampleAt?.(samples, clock) ?? greenSample(clock)
|
||||
samples += 1
|
||||
if (result instanceof Error) throw result
|
||||
return result
|
||||
},
|
||||
readHardRules: async () => {
|
||||
const result = input.hardRulesAt?.(reads) ?? calm
|
||||
reads += 1
|
||||
if (result instanceof Error) throw result
|
||||
return result
|
||||
}
|
||||
})
|
||||
return { run, startedAt: clock, samples: () => samples }
|
||||
}
|
||||
|
||||
describe('pre-drain sample window', () => {
|
||||
it('sizes the window to the hosts the drain will re-place', () => {
|
||||
expect(preDrainSampleWindowMinutes(0)).toBe(3)
|
||||
expect(preDrainSampleWindowMinutes(150)).toBe(3)
|
||||
expect(preDrainSampleWindowMinutes(500)).toBe(3)
|
||||
expect(preDrainSampleWindowMinutes(501)).toBe(5)
|
||||
expect(preDrainSampleWindowMinutes(1_500)).toBe(5)
|
||||
expect(preDrainSampleWindowMinutes(1_501)).toBe(8)
|
||||
expect(preDrainSampleWindowMinutes(2_750)).toBe(8)
|
||||
})
|
||||
|
||||
it('refuses a host count it cannot size', () => {
|
||||
for (const hosts of [-1, 1.5, Number.NaN, Number.POSITIVE_INFINITY]) {
|
||||
expect(() => preDrainSampleWindowMinutes(hosts)).toThrow('host count is invalid')
|
||||
}
|
||||
})
|
||||
})
|
||||
|
||||
describe('cell exit attribution', () => {
|
||||
const scope = {
|
||||
targetCellId: 'production-gce-c25',
|
||||
placementCellIds: new Set(['production-gce-c25', 'production-gce-c26', 'production-gce-c17']),
|
||||
configuredCellIds: new Set([
|
||||
'production-gce-c5', 'production-gce-c17', 'production-gce-c25', 'production-gce-c26'
|
||||
])
|
||||
}
|
||||
const attribute = (exits: Record<string, number>, cells: Record<string, string | null>) =>
|
||||
attributeCellExits({
|
||||
exitsByInstance: new Map(Object.entries(exits)),
|
||||
cellByInstance: new Map(Object.entries(cells)),
|
||||
...scope
|
||||
})
|
||||
|
||||
it('ignores the target cell rolling the fix for its own crashes', () => {
|
||||
expect(attribute({ '25': 12 }, { '25': 'production-gce-c25' }))
|
||||
.toEqual({ counted: 0, unattributed: [] })
|
||||
})
|
||||
|
||||
it('counts an exit on another general or migration-only cell', () => {
|
||||
expect(attribute({ '26': 1 }, { '26': 'production-gce-c26' }))
|
||||
.toEqual({ counted: 1, unattributed: [] })
|
||||
expect(attribute({ '17': 2 }, { '17': 'production-gce-c17' }))
|
||||
.toEqual({ counted: 2, unattributed: [] })
|
||||
})
|
||||
|
||||
it('ignores an existing-only legacy cell, which takes no placements', () => {
|
||||
expect(attribute({ '5': 15 }, { '5': 'production-gce-c5' }))
|
||||
.toEqual({ counted: 0, unattributed: [] })
|
||||
})
|
||||
|
||||
it('reports an instance with no cell, or an unknown cell, as unattributed', () => {
|
||||
expect(attribute({ '9': 1, '8': 1 }, { '9': null, '8': 'production-gce-c99' }))
|
||||
.toEqual({ counted: 0, unattributed: ['8', '9'] })
|
||||
expect(attribute({ '7': 1 }, {})).toEqual({ counted: 0, unattributed: ['7'] })
|
||||
})
|
||||
|
||||
it('trips the rule through the evaluator for a counted exit', () => {
|
||||
const { counted } = attribute({ '26': 1, '25': 3, '5': 4 }, {
|
||||
'26': 'production-gce-c26', '25': 'production-gce-c25', '5': 'production-gce-c5'
|
||||
})
|
||||
expect(evaluatePreDrainHardRules({ ...calm, cellProcessExits: counted })).toEqual([
|
||||
expect.objectContaining({ signal: 'cells.process_exits_10m', observed: 1 })
|
||||
])
|
||||
})
|
||||
})
|
||||
|
||||
describe('pre-drain hard rules', () => {
|
||||
it('passes a calm fleet', () => {
|
||||
expect(evaluatePreDrainHardRules(calm)).toEqual([])
|
||||
})
|
||||
|
||||
it('trips on any cell container exit in the lookback', () => {
|
||||
expect(evaluatePreDrainHardRules({ ...calm, cellProcessExits: 1 })).toEqual([
|
||||
expect.objectContaining({ signal: 'cells.process_exits_10m', observed: 1, threshold: 0 })
|
||||
])
|
||||
})
|
||||
|
||||
it('trips on an exit no cell could be named for', () => {
|
||||
expect(evaluatePreDrainHardRules({ ...calm, unattributedExitInstances: ['111', '222'] }))
|
||||
.toEqual([expect.objectContaining({
|
||||
code: 'exit_unattributed',
|
||||
signal: 'cells.process_exits_10m.instances=111+222',
|
||||
observed: 2
|
||||
})])
|
||||
})
|
||||
|
||||
it('trips on a disconnect pulse, not on a busy healthy minute', () => {
|
||||
expect(evaluatePreDrainHardRules({ ...calm, director503PeakPerMinute: 500 })).toEqual([])
|
||||
expect(evaluatePreDrainHardRules({ ...calm, director503PeakPerMinute: 501 })).toEqual([
|
||||
expect.objectContaining({ signal: 'director.503_peak_minute_10m', observed: 501 })
|
||||
])
|
||||
})
|
||||
|
||||
it('holds director concurrency p99 to the monitor threshold', () => {
|
||||
expect(PRE_DRAIN_HARD_RULES.directorConcurrencyP99Max)
|
||||
.toBe(INCIDENT_MONITOR_THRESHOLDS.directorConcurrency)
|
||||
const max = PRE_DRAIN_HARD_RULES.directorConcurrencyP99Max
|
||||
expect(evaluatePreDrainHardRules({ ...calm, directorConcurrencyP99: max })).toEqual([])
|
||||
// The 2026-10-01 13:42Z refusal.
|
||||
expect(evaluatePreDrainHardRules({ ...calm, directorConcurrencyP99: 66.8 })).toEqual([
|
||||
expect.objectContaining({ code: 'threshold_max', signal: 'director.concurrency_p99_4m' })
|
||||
])
|
||||
})
|
||||
|
||||
it('reads an unpublished concurrency series as missing, never as calm', () => {
|
||||
expect(evaluatePreDrainHardRules({ ...calm, directorConcurrencyP99: null })).toEqual([
|
||||
expect.objectContaining({ code: 'signal_missing', signal: 'director.concurrency_p99_4m' })
|
||||
])
|
||||
})
|
||||
})
|
||||
|
||||
describe('runPreDrainSample', () => {
|
||||
it('passes after one sample per minute across the whole window', async () => {
|
||||
for (const windowMinutes of [3, 5, 8]) {
|
||||
const { run, startedAt, samples } = harness({ windowMinutes })
|
||||
const result = await run
|
||||
expect(samples()).toBe(windowMinutes + 1)
|
||||
expect(Date.parse(result.completedAt) - startedAt).toBeGreaterThanOrEqual(
|
||||
windowMinutes * 60_000
|
||||
)
|
||||
}
|
||||
})
|
||||
|
||||
it('refuses a window that is not one of the sized bands', async () => {
|
||||
await expect(harness({ windowMinutes: 15 }).run).rejects.toThrow('window is invalid')
|
||||
})
|
||||
|
||||
it('trips at once on a hard rule, before the window ends', async () => {
|
||||
const { run, samples } = harness({
|
||||
windowMinutes: 8,
|
||||
hardRulesAt: (index) => (index === 2 ? { ...calm, cellProcessExits: 1 } : calm)
|
||||
})
|
||||
await expect(run).rejects.toBeInstanceOf(PreDrainSampleTripped)
|
||||
await expect(run).rejects.toThrow('cells.process_exits_10m')
|
||||
expect(samples()).toBe(3)
|
||||
})
|
||||
|
||||
it('trips at once on a monitor threshold the monitor never tolerates', async () => {
|
||||
const { run } = harness({
|
||||
windowMinutes: 3,
|
||||
sampleAt: (index, at) => {
|
||||
const next = greenSample(at)
|
||||
if (index === 1) {
|
||||
next.sources['cloud-monitoring']!.signals['director.concurrency'] = {
|
||||
value: 80,
|
||||
observedAt: new Date(at).toISOString()
|
||||
}
|
||||
}
|
||||
return next
|
||||
}
|
||||
})
|
||||
await expect(run).rejects.toThrow('director.concurrency observed=80')
|
||||
})
|
||||
|
||||
it('rides out a cell probe within the monitor tolerance and trips past it', async () => {
|
||||
const failingProbe = (failFor: number) => (index: number, at: number) => {
|
||||
const next = greenSample(at)
|
||||
if (index >= 1 && index < 1 + failFor) {
|
||||
next.sources['active-probe']!.signals['cell.production-gce-c1.health'] = {
|
||||
value: 0,
|
||||
observedAt: new Date(at).toISOString()
|
||||
}
|
||||
}
|
||||
return next
|
||||
}
|
||||
const tolerance = INCIDENT_MONITOR_THRESHOLDS.cellProbeToleranceSamples
|
||||
await expect(
|
||||
harness({ windowMinutes: 3, sampleAt: failingProbe(tolerance) }).run
|
||||
).resolves.toBeDefined()
|
||||
await expect(
|
||||
harness({ windowMinutes: 3, sampleAt: failingProbe(tolerance + 1) }).run
|
||||
).rejects.toThrow('cell.production-gce-c1.health')
|
||||
})
|
||||
|
||||
it('rides out failed reads within the freshness tolerance and trips past it', async () => {
|
||||
const failingReads = (failFor: number) => (index: number) =>
|
||||
index >= 1 && index < 1 + failFor ? new Error('Cloud Monitoring returned 503') : calm
|
||||
await expect(
|
||||
harness({
|
||||
windowMinutes: 3,
|
||||
hardRulesAt: failingReads(INCIDENT_FRESHNESS_TOLERANCE_SAMPLES)
|
||||
}).run
|
||||
).resolves.toBeDefined()
|
||||
await expect(
|
||||
harness({
|
||||
windowMinutes: 3,
|
||||
hardRulesAt: failingReads(INCIDENT_FRESHNESS_TOLERANCE_SAMPLES + 1)
|
||||
}).run
|
||||
).rejects.toThrow('collector_failed pre-drain.hard-rules')
|
||||
})
|
||||
|
||||
// Why: the drain starts right after this returns, so it must not start on a reading the
|
||||
// tolerance was still carrying.
|
||||
it('keeps sampling past the window until a sample is clean', async () => {
|
||||
const { run, samples } = harness({
|
||||
windowMinutes: 3,
|
||||
hardRulesAt: (index) => (index === 3 ? { ...calm, directorConcurrencyP99: null } : calm)
|
||||
})
|
||||
const result = await run
|
||||
expect(samples()).toBe(5)
|
||||
expect(result.samples.at(-1)?.failures).toEqual([])
|
||||
})
|
||||
|
||||
// Why: two tolerated readings that alternate never build a streak, so only the overrun cap
|
||||
// stops them from holding the step open until its timeout.
|
||||
it('trips when alternating tolerated readings never leave a clean sample', async () => {
|
||||
const { run, samples } = harness({
|
||||
windowMinutes: 3,
|
||||
hardRulesAt: (index) =>
|
||||
index >= 3 && index % 2 === 1 ? { ...calm, directorConcurrencyP99: null } : calm,
|
||||
sampleAt: (index, at) =>
|
||||
index >= 3 && index % 2 === 0 ? new Error('collector down') : greenSample(at)
|
||||
})
|
||||
await expect(run).rejects.toBeInstanceOf(PreDrainSampleTripped)
|
||||
expect(samples()).toBe(4 + PRE_DRAIN_MAX_OVERRUN_SAMPLES)
|
||||
})
|
||||
|
||||
it('records why a read failed', async () => {
|
||||
const records: string[][] = []
|
||||
let clock = 0
|
||||
await runPreDrainSample({
|
||||
windowMinutes: 3,
|
||||
now: () => clock,
|
||||
wait: async (ms) => { clock += ms },
|
||||
collect: async () => greenSample(clock),
|
||||
readHardRules: async () => {
|
||||
if (records.length === 1) throw new Error('Google telemetry returned 403')
|
||||
return calm
|
||||
},
|
||||
log: (record) => records.push(record.errors)
|
||||
})
|
||||
expect(records[1]).toEqual(['hard rules: Google telemetry returned 403'])
|
||||
})
|
||||
})
|
||||
@@ -1,259 +0,0 @@
|
||||
import {
|
||||
describeIncidentFailure,
|
||||
evaluateIncidentSample,
|
||||
FRESHNESS_FAILURE_CODES,
|
||||
INCIDENT_FRESHNESS_TOLERANCE_SAMPLES,
|
||||
INCIDENT_MONITOR_THRESHOLDS,
|
||||
toleratedStreakKey,
|
||||
type IncidentFailure,
|
||||
type IncidentSample
|
||||
} from './incident-monitor.js'
|
||||
|
||||
// Inline fleet-health sample a same-cap wave takes right before it isolates its cell. It judges
|
||||
// every sample with the monitor's own evaluator and tolerances; what it adds is a window sized to
|
||||
// the drain and three lookback rules no single sample can see.
|
||||
|
||||
export const PRE_DRAIN_SAMPLE_INTERVAL_MS = 60_000
|
||||
|
||||
// A bigger drain re-places more hosts through the director's single sticky lane, so it gets a
|
||||
// longer quiet window before it starts. Ordered; the first band whose ceiling covers the cell wins.
|
||||
export const PRE_DRAIN_SAMPLE_WINDOWS = [
|
||||
{ maxHosts: 500, minutes: 3 },
|
||||
{ maxHosts: 1_500, minutes: 5 },
|
||||
{ maxHosts: Number.POSITIVE_INFINITY, minutes: 8 }
|
||||
] as const
|
||||
|
||||
export const PRE_DRAIN_HARD_RULES = {
|
||||
// Any container exit on a cell that takes placements, other than the cell being rolled; a drain
|
||||
// into a crash loop re-places its hosts twice.
|
||||
cellProcessExitLookbackMs: 10 * 60_000,
|
||||
cellProcessExitsMax: 0,
|
||||
// A disconnect pulse lands as thousands of director 503s in one minute (7,750 and 13,000 on
|
||||
// 2026-09-20 and 2026-10-01) against a healthy 20-70.
|
||||
director503LookbackMs: 10 * 60_000,
|
||||
director503PerMinuteMax: 500,
|
||||
// The live preflight reads only the newest concurrency point; this holds the last four minutes.
|
||||
directorConcurrencyLookbackMs: 4 * 60_000,
|
||||
directorConcurrencyP99Max: INCIDENT_MONITOR_THRESHOLDS.directorConcurrency
|
||||
} as const
|
||||
|
||||
export type PreDrainHardRuleReadings = {
|
||||
// Exits on general or migration-only cells other than the target.
|
||||
cellProcessExits: number
|
||||
// Exits on an instance no cell could be resolved for; never assumed harmless.
|
||||
unattributedExitInstances: string[]
|
||||
director503PeakPerMinute: number
|
||||
// Null when Cloud Monitoring published no point in the lookback.
|
||||
directorConcurrencyP99: number | null
|
||||
}
|
||||
|
||||
// Which exits count. The target's own exits are why it is being rolled (c25 crashed 12 times in
|
||||
// a week and is the first US roll), and existing-only legacy cells take no placements (c5 alone
|
||||
// exited 15 times in that week), so neither can stop a drain. An instance with no known cell
|
||||
// counts as unattributed and trips the rule.
|
||||
export function attributeCellExits(input: {
|
||||
exitsByInstance: ReadonlyMap<string, number>
|
||||
cellByInstance: ReadonlyMap<string, string | null>
|
||||
targetCellId: string
|
||||
placementCellIds: ReadonlySet<string>
|
||||
configuredCellIds: ReadonlySet<string>
|
||||
}): { counted: number; unattributed: string[] } {
|
||||
let counted = 0
|
||||
const unattributed: string[] = []
|
||||
for (const [instanceId, exits] of input.exitsByInstance) {
|
||||
if (exits <= 0) continue
|
||||
const cellId = input.cellByInstance.get(instanceId) ?? null
|
||||
if (cellId === null || !input.configuredCellIds.has(cellId)) {
|
||||
unattributed.push(instanceId)
|
||||
continue
|
||||
}
|
||||
if (cellId === input.targetCellId || !input.placementCellIds.has(cellId)) continue
|
||||
counted += exits
|
||||
}
|
||||
return { counted, unattributed: unattributed.sort() }
|
||||
}
|
||||
|
||||
export function preDrainSampleWindowMinutes(hostCount: number): number {
|
||||
if (!Number.isSafeInteger(hostCount) || hostCount < 0) {
|
||||
throw new Error('pre-drain sample host count is invalid')
|
||||
}
|
||||
const band = PRE_DRAIN_SAMPLE_WINDOWS.find((entry) => hostCount <= entry.maxHosts)
|
||||
if (!band) throw new Error('pre-drain sample host count is invalid')
|
||||
return band.minutes
|
||||
}
|
||||
|
||||
export function evaluatePreDrainHardRules(readings: PreDrainHardRuleReadings): IncidentFailure[] {
|
||||
const failures: IncidentFailure[] = []
|
||||
if (readings.cellProcessExits > PRE_DRAIN_HARD_RULES.cellProcessExitsMax) {
|
||||
failures.push({
|
||||
code: 'threshold_max',
|
||||
source: 'cloud-monitoring',
|
||||
signal: 'cells.process_exits_10m',
|
||||
observed: readings.cellProcessExits,
|
||||
threshold: PRE_DRAIN_HARD_RULES.cellProcessExitsMax
|
||||
})
|
||||
}
|
||||
if (readings.unattributedExitInstances.length > 0) {
|
||||
failures.push({
|
||||
code: 'exit_unattributed',
|
||||
source: 'cloud-monitoring',
|
||||
signal: `cells.process_exits_10m.instances=${readings.unattributedExitInstances.join('+')}`,
|
||||
observed: readings.unattributedExitInstances.length,
|
||||
threshold: 0
|
||||
})
|
||||
}
|
||||
if (readings.director503PeakPerMinute > PRE_DRAIN_HARD_RULES.director503PerMinuteMax) {
|
||||
failures.push({
|
||||
code: 'threshold_max',
|
||||
source: 'cloud-monitoring',
|
||||
signal: 'director.503_peak_minute_10m',
|
||||
observed: readings.director503PeakPerMinute,
|
||||
threshold: PRE_DRAIN_HARD_RULES.director503PerMinuteMax
|
||||
})
|
||||
}
|
||||
if (readings.directorConcurrencyP99 === null) {
|
||||
failures.push({
|
||||
code: 'signal_missing',
|
||||
source: 'cloud-monitoring',
|
||||
signal: 'director.concurrency_p99_4m'
|
||||
})
|
||||
} else if (readings.directorConcurrencyP99 > PRE_DRAIN_HARD_RULES.directorConcurrencyP99Max) {
|
||||
failures.push({
|
||||
code: 'threshold_max',
|
||||
source: 'cloud-monitoring',
|
||||
signal: 'director.concurrency_p99_4m',
|
||||
observed: readings.directorConcurrencyP99,
|
||||
threshold: PRE_DRAIN_HARD_RULES.directorConcurrencyP99Max
|
||||
})
|
||||
}
|
||||
return failures
|
||||
}
|
||||
|
||||
export type PreDrainSampleRecord = {
|
||||
at: string
|
||||
failures: IncidentFailure[]
|
||||
// Why a read failed; a failure code alone hides a permission error or a bad filter.
|
||||
errors: string[]
|
||||
}
|
||||
|
||||
// Samples past the window that may still carry a tolerated failure; alternating tolerated readings
|
||||
// (cell X, then cell Y) never trip a streak, so this is what bounds the overrun.
|
||||
export const PRE_DRAIN_MAX_OVERRUN_SAMPLES = INCIDENT_FRESHNESS_TOLERANCE_SAMPLES + 1
|
||||
|
||||
export type PreDrainSampleResult = {
|
||||
windowMinutes: number
|
||||
startedAt: string
|
||||
completedAt: string
|
||||
samples: PreDrainSampleRecord[]
|
||||
}
|
||||
|
||||
export class PreDrainSampleTripped extends Error {
|
||||
constructor(readonly failures: IncidentFailure[], readonly samples: PreDrainSampleRecord[]) {
|
||||
super(`relay pre-drain sample tripped: ${failures.map(describeIncidentFailure).join(',')}`)
|
||||
}
|
||||
}
|
||||
|
||||
const CONTINUITY_CODES = new Set(['collector_failed', ...FRESHNESS_FAILURE_CODES])
|
||||
|
||||
function continuityKey(failure: IncidentFailure): string {
|
||||
return `${failure.source}/${failure.signal ?? '*'}`
|
||||
}
|
||||
|
||||
function bumpStreaks(streaks: Map<string, number>, keys: Set<string>): Map<string, number> {
|
||||
const next = new Map<string, number>()
|
||||
for (const key of keys) next.set(key, (streaks.get(key) ?? 0) + 1)
|
||||
return next
|
||||
}
|
||||
|
||||
/**
|
||||
* Samples once a minute until the window has elapsed, then keeps going until a sample comes back
|
||||
* with no failure at all, so the drain never starts on an open reading. A breach of any
|
||||
* non-tolerated threshold or hard rule trips at once; readings the monitor tolerates (a cell probe
|
||||
* or the director instance count, an unread signal, a failed collector round trip) trip only past
|
||||
* the monitor's own consecutive-sample budget. The overrun is capped separately.
|
||||
*/
|
||||
export async function runPreDrainSample(input: {
|
||||
windowMinutes: number
|
||||
collect: () => Promise<IncidentSample>
|
||||
readHardRules: () => Promise<PreDrainHardRuleReadings>
|
||||
now?: () => number
|
||||
wait?: (ms: number) => Promise<void>
|
||||
log?: (record: PreDrainSampleRecord) => void
|
||||
}): Promise<PreDrainSampleResult> {
|
||||
if (!PRE_DRAIN_SAMPLE_WINDOWS.some((band) => band.minutes === input.windowMinutes)) {
|
||||
throw new Error('pre-drain sample window is invalid')
|
||||
}
|
||||
const now = input.now ?? Date.now
|
||||
const wait = input.wait ?? ((ms: number) => new Promise<void>((resolveWait) => {
|
||||
setTimeout(resolveWait, ms)
|
||||
}))
|
||||
const startedMs = now()
|
||||
const windowMs = input.windowMinutes * 60_000
|
||||
const samples: PreDrainSampleRecord[] = []
|
||||
let continuityStreaks = new Map<string, number>()
|
||||
let probeStreaks = new Map<string, number>()
|
||||
let overrunSamples = 0
|
||||
for (;;) {
|
||||
const sampleStartedMs = now()
|
||||
const failures: IncidentFailure[] = []
|
||||
const errors: string[] = []
|
||||
const reason = (error: unknown) => error instanceof Error ? error.message : String(error)
|
||||
try {
|
||||
failures.push(
|
||||
...evaluateIncidentSample(await input.collect(), now(), 'strict', null, null).failures
|
||||
)
|
||||
} catch (error) {
|
||||
failures.push({ code: 'collector_failed', source: 'cloud-monitoring' })
|
||||
errors.push(`collector: ${reason(error)}`)
|
||||
}
|
||||
try {
|
||||
failures.push(...evaluatePreDrainHardRules(await input.readHardRules()))
|
||||
} catch (error) {
|
||||
failures.push({
|
||||
code: 'collector_failed',
|
||||
source: 'cloud-monitoring',
|
||||
signal: 'pre-drain.hard-rules'
|
||||
})
|
||||
errors.push(`hard rules: ${reason(error)}`)
|
||||
}
|
||||
const record = { at: new Date(now()).toISOString(), failures, errors }
|
||||
samples.push(record)
|
||||
input.log?.(record)
|
||||
const continuity = failures.filter((failure) => CONTINUITY_CODES.has(failure.code))
|
||||
const breaches = failures.filter((failure) => !CONTINUITY_CODES.has(failure.code))
|
||||
continuityStreaks = bumpStreaks(continuityStreaks, new Set(continuity.map(continuityKey)))
|
||||
const probeKeys = new Set<string>()
|
||||
for (const failure of breaches) {
|
||||
const key = toleratedStreakKey(failure)
|
||||
if (key !== null) probeKeys.add(key)
|
||||
}
|
||||
probeStreaks = bumpStreaks(probeStreaks, probeKeys)
|
||||
const tripped = [
|
||||
...breaches.filter((failure) => {
|
||||
const key = toleratedStreakKey(failure)
|
||||
return key === null ||
|
||||
(probeStreaks.get(key) ?? 0) > INCIDENT_MONITOR_THRESHOLDS.cellProbeToleranceSamples
|
||||
}),
|
||||
...continuity.filter((failure) =>
|
||||
(continuityStreaks.get(continuityKey(failure)) ?? 0) >
|
||||
INCIDENT_FRESHNESS_TOLERANCE_SAMPLES)
|
||||
]
|
||||
if (tripped.length > 0) throw new PreDrainSampleTripped(tripped, samples)
|
||||
const windowDone = now() - startedMs >= windowMs
|
||||
if (windowDone && failures.length > 0) {
|
||||
overrunSamples += 1
|
||||
if (overrunSamples > PRE_DRAIN_MAX_OVERRUN_SAMPLES) {
|
||||
throw new PreDrainSampleTripped(failures, samples)
|
||||
}
|
||||
}
|
||||
if (windowDone && failures.length === 0) {
|
||||
return {
|
||||
windowMinutes: input.windowMinutes,
|
||||
startedAt: new Date(startedMs).toISOString(),
|
||||
completedAt: record.at,
|
||||
samples
|
||||
}
|
||||
}
|
||||
await wait(Math.max(0, sampleStartedMs + PRE_DRAIN_SAMPLE_INTERVAL_MS - now()))
|
||||
}
|
||||
}
|
||||
@@ -1,42 +0,0 @@
|
||||
import { readdirSync, readFileSync } from 'node:fs'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import {
|
||||
RELAY_GITHUB_REPOSITORY,
|
||||
RELAY_WORKFLOW_FILE_PREFIX,
|
||||
relayRepositoryApiPath,
|
||||
relayWorkflowFile
|
||||
} from './relay-repository.js'
|
||||
|
||||
const sourceDir = fileURLToPath(new URL('.', import.meta.url))
|
||||
const sources = readdirSync(sourceDir)
|
||||
.filter((name) => name.endsWith('.ts') && name !== 'relay-repository.ts')
|
||||
.map((name) => ({ name, text: readFileSync(`${sourceDir}${name}`, 'utf8') }))
|
||||
|
||||
describe('relay repository identity', () => {
|
||||
it('builds API paths and workflow filenames from the one repository name', () => {
|
||||
expect(relayRepositoryApiPath('actions/runs')).toBe(`repos/${RELAY_GITHUB_REPOSITORY}/actions/runs`)
|
||||
expect(relayWorkflowFile('power-relay-staging.yml')).toBe(
|
||||
`${RELAY_WORKFLOW_FILE_PREFIX}power-relay-staging.yml`
|
||||
)
|
||||
})
|
||||
|
||||
// Why: the public-repo copy renames the repository and prefixes every workflow file. Both have to
|
||||
// be one edit, so no other module may restate either.
|
||||
it('is the only module naming a GitHub repository', () => {
|
||||
for (const { name, text } of sources) {
|
||||
expect(text, `${name} restates a GitHub repository`).not.toMatch(/stablyai\//)
|
||||
}
|
||||
})
|
||||
|
||||
it('is the only module naming a workflow file', () => {
|
||||
for (const { name, text } of sources) {
|
||||
for (const match of text.matchAll(/'([^']*\.yml)'/g)) {
|
||||
const file = match[1] ?? ''
|
||||
expect(text, `${name} names ${file} outside relayWorkflowFile`).toMatch(
|
||||
new RegExp(`relayWorkflowFile\\('${file.replaceAll('.', '\\.')}'\\)`)
|
||||
)
|
||||
}
|
||||
}
|
||||
})
|
||||
})
|
||||
@@ -1,14 +0,0 @@
|
||||
// Single place naming the GitHub repository that holds the Relay workflows. When the Relay tree is
|
||||
// copied to its public repository, only this file changes: the repository moves and every workflow
|
||||
// file gains a prefix, while the workflow display names stay as they are.
|
||||
export const RELAY_GITHUB_REPOSITORY = 'stablyai/orca-cloud'
|
||||
|
||||
export const RELAY_WORKFLOW_FILE_PREFIX = ''
|
||||
|
||||
export function relayWorkflowFile(name: string): string {
|
||||
return `${RELAY_WORKFLOW_FILE_PREFIX}${name}`
|
||||
}
|
||||
|
||||
export function relayRepositoryApiPath(resource: string): string {
|
||||
return `repos/${RELAY_GITHUB_REPOSITORY}/${resource}`
|
||||
}
|
||||
@@ -1,370 +0,0 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { RELAY_OPS_ENVIRONMENTS } from './environment-config.js'
|
||||
import type { GcloudClient } from './gcloud-client.js'
|
||||
import { probeEndpointHealth, readResourceInventory } from './resource-inventory.js'
|
||||
|
||||
const digest = `sha256:${'a'.repeat(64)}`
|
||||
const runService = {
|
||||
template: {
|
||||
scaling: { minInstanceCount: 0, maxInstanceCount: 2 },
|
||||
containers: [{ image: 'registry/image:tag' }]
|
||||
},
|
||||
conditions: [{ state: 'CONDITION_SUCCEEDED' }],
|
||||
latestReadyRevision: 'projects/project/revisions/revision-one'
|
||||
}
|
||||
|
||||
const sleepingStagingGcloud: GcloudClient = { accessToken: async () => 'a'.repeat(40) }
|
||||
|
||||
// Staging's Cloud SQL is stopped, so this inventory reads REST only and probes no endpoint.
|
||||
type MigOutcome = 'ok' | 'throw' | 'missing'
|
||||
const sleepingStagingFetch = (migOutcome: (migName: string) => MigOutcome): typeof fetch =>
|
||||
async (input) => {
|
||||
const url = new URL(String(input))
|
||||
if (url.hostname === 'run.googleapis.com') return Response.json(runService)
|
||||
if (url.hostname === 'sqladmin.googleapis.com') return Response.json({
|
||||
state: 'STOPPED',
|
||||
databaseVersion: 'POSTGRES_17',
|
||||
settings: { activationPolicy: 'NEVER', availabilityType: 'ZONAL', tier: 'db-custom-1-3840' }
|
||||
})
|
||||
if (url.hostname === 'certificatemanager.googleapis.com') return Response.json({
|
||||
managed: { domains: ['*.relay-staging.onorca.dev'], state: 'ACTIVE' }
|
||||
})
|
||||
if (url.pathname.includes('/instanceGroupManagers/')) {
|
||||
const name = url.pathname.split('/').at(-1)!
|
||||
const outcome = migOutcome(name)
|
||||
if (outcome === 'throw') throw new TypeError('fetch failed')
|
||||
if (outcome === 'missing') return new Response(null, { status: 404 })
|
||||
return Response.json({
|
||||
name,
|
||||
targetSize: 0,
|
||||
size: '0',
|
||||
instanceGroup: `projects/project/zones/zone/instanceGroups/${name}`,
|
||||
instanceTemplate: `projects/project/global/instanceTemplates/template-${name}`,
|
||||
status: { isStable: true }
|
||||
})
|
||||
}
|
||||
if (url.pathname.includes('/instanceTemplates/')) return Response.json({ properties: {} })
|
||||
if (url.pathname.endsWith('/getHealth')) return Response.json([])
|
||||
throw new Error(`Unexpected request to ${url.hostname}${url.pathname}`)
|
||||
}
|
||||
|
||||
describe('readResourceInventory', () => {
|
||||
it('does not delay a healthy endpoint sample', async () => {
|
||||
let calls = 0
|
||||
let waits = 0
|
||||
const result = await probeEndpointHealth(
|
||||
'https://c9.relay.onorca.dev',
|
||||
async () => {
|
||||
calls += 1
|
||||
return new Response(null, { status: 200 })
|
||||
},
|
||||
{
|
||||
wait: async () => {
|
||||
waits += 1
|
||||
}
|
||||
}
|
||||
)
|
||||
|
||||
expect(result.health).toBe(true)
|
||||
expect(result.ready).toBe(true)
|
||||
expect(calls).toBe(2)
|
||||
expect(waits).toBe(0)
|
||||
})
|
||||
|
||||
it('retries one transient endpoint failure within the same sample', async () => {
|
||||
const calls = new Map<string, number>()
|
||||
const waits: number[] = []
|
||||
const result = await probeEndpointHealth(
|
||||
'https://c9.relay.onorca.dev',
|
||||
async (input) => {
|
||||
const path = new URL(String(input)).pathname
|
||||
const call = (calls.get(path) ?? 0) + 1
|
||||
calls.set(path, call)
|
||||
return new Response(null, { status: path === '/ready' && call === 1 ? 503 : 200 })
|
||||
},
|
||||
{
|
||||
wait: async (ms) => {
|
||||
waits.push(ms)
|
||||
}
|
||||
}
|
||||
)
|
||||
|
||||
expect(result.health).toBe(true)
|
||||
expect(result.ready).toBe(true)
|
||||
expect(calls).toEqual(new Map([['/health', 2], ['/ready', 2]]))
|
||||
expect(waits).toEqual([11_000])
|
||||
})
|
||||
|
||||
it('fails closed when the endpoint retry is also unhealthy', async () => {
|
||||
let calls = 0
|
||||
const waits: number[] = []
|
||||
const result = await probeEndpointHealth(
|
||||
'https://c9.relay.onorca.dev',
|
||||
async () => {
|
||||
calls += 1
|
||||
return new Response(null, { status: 503 })
|
||||
},
|
||||
{
|
||||
wait: async (ms) => {
|
||||
waits.push(ms)
|
||||
}
|
||||
}
|
||||
)
|
||||
|
||||
expect(result.health).toBe(false)
|
||||
expect(result.ready).toBe(false)
|
||||
expect(calls).toBe(4)
|
||||
// A refusing endpoint is a reading, so only the independent retry runs.
|
||||
expect(waits).toEqual([11_000])
|
||||
})
|
||||
|
||||
it('treats a thrown fetch as no reading and re-asks that path once', async () => {
|
||||
const calls: string[] = []
|
||||
const waits: number[] = []
|
||||
const result = await probeEndpointHealth(
|
||||
'https://c9.relay.onorca.dev',
|
||||
async (input) => {
|
||||
const path = new URL(String(input)).pathname
|
||||
calls.push(path)
|
||||
if (path === '/health' && calls.filter((call) => call === '/health').length === 1) {
|
||||
throw new TypeError('fetch failed')
|
||||
}
|
||||
return new Response(null, { status: 200 })
|
||||
},
|
||||
{
|
||||
wait: async (ms) => {
|
||||
waits.push(ms)
|
||||
}
|
||||
}
|
||||
)
|
||||
|
||||
expect(result.health).toBe(true)
|
||||
expect(result.ready).toBe(true)
|
||||
expect(calls.filter((call) => call === '/health')).toEqual(['/health', '/health'])
|
||||
expect(waits).toEqual([1_000])
|
||||
})
|
||||
|
||||
it('fails closed when both attempts of a path throw', async () => {
|
||||
const calls: string[] = []
|
||||
const waits: number[] = []
|
||||
const result = await probeEndpointHealth(
|
||||
'https://c9.relay.onorca.dev',
|
||||
async (input) => {
|
||||
const path = new URL(String(input)).pathname
|
||||
calls.push(path)
|
||||
if (path === '/health') throw new TypeError('fetch failed')
|
||||
return new Response(null, { status: 200 })
|
||||
},
|
||||
{
|
||||
wait: async (ms) => {
|
||||
waits.push(ms)
|
||||
}
|
||||
}
|
||||
)
|
||||
|
||||
expect(result.health).toBe(false)
|
||||
expect(calls.filter((call) => call === '/health')).toHaveLength(4)
|
||||
expect(waits).toEqual([1_000, 11_000, 1_000])
|
||||
})
|
||||
|
||||
it('accepts an auth-shaped endpoint that serves no readiness path', async () => {
|
||||
const calls: string[] = []
|
||||
const waits: number[] = []
|
||||
const result = await probeEndpointHealth(
|
||||
'https://login.onorca.dev',
|
||||
async (input) => {
|
||||
const path = new URL(String(input)).pathname
|
||||
calls.push(path)
|
||||
return new Response(null, { status: path === '/ready' ? 404 : 200 })
|
||||
},
|
||||
{
|
||||
requiresReady: false,
|
||||
wait: async (ms) => {
|
||||
waits.push(ms)
|
||||
}
|
||||
}
|
||||
)
|
||||
|
||||
expect(result.health).toBe(true)
|
||||
expect(result.ready).toBeNull()
|
||||
expect(calls).toEqual(['/health'])
|
||||
expect(waits).toEqual([])
|
||||
})
|
||||
|
||||
it('still requires readiness for the director and cells', async () => {
|
||||
const waits: number[] = []
|
||||
const result = await probeEndpointHealth(
|
||||
'https://relay.onorca.dev',
|
||||
async (input) => new Response(null, {
|
||||
status: new URL(String(input)).pathname === '/ready' ? 503 : 200
|
||||
}),
|
||||
{
|
||||
wait: async (ms) => {
|
||||
waits.push(ms)
|
||||
}
|
||||
}
|
||||
)
|
||||
|
||||
expect(result.health).toBe(true)
|
||||
expect(result.ready).toBe(false)
|
||||
expect(waits).toEqual([11_000])
|
||||
})
|
||||
|
||||
it('measures latency as the answering round trip, not the retry delay', async () => {
|
||||
let healthCalls = 0
|
||||
const result = await probeEndpointHealth(
|
||||
'https://c9.relay.onorca.dev',
|
||||
async (input) => {
|
||||
if (new URL(String(input)).pathname !== '/health') return new Response(null, { status: 200 })
|
||||
healthCalls += 1
|
||||
if (healthCalls === 1) throw new TypeError('fetch failed')
|
||||
return new Response(null, { status: 200 })
|
||||
},
|
||||
{ wait: async (ms) => await new Promise((resolve) => setTimeout(resolve, Math.min(ms, 60))) }
|
||||
)
|
||||
|
||||
expect(result.health).toBe(true)
|
||||
expect(result.latencyMs).not.toBeNull()
|
||||
expect(result.latencyMs!).toBeLessThan(60)
|
||||
})
|
||||
|
||||
it('uses aggregate REST inventory without probing sleeping staging endpoints', async () => {
|
||||
const gcloud: GcloudClient = { accessToken: async () => 'a'.repeat(40) }
|
||||
let publicProbeCalls = 0
|
||||
const fetchImpl: typeof fetch = async (input) => {
|
||||
const url = new URL(String(input))
|
||||
if (url.hostname.endsWith('onorca.dev')) {
|
||||
publicProbeCalls += 1
|
||||
return Response.json({ status: 'ok' })
|
||||
}
|
||||
if (url.hostname === 'run.googleapis.com') return Response.json(runService)
|
||||
if (url.hostname === 'sqladmin.googleapis.com') return Response.json({
|
||||
state: 'STOPPED',
|
||||
databaseVersion: 'POSTGRES_17',
|
||||
settings: {
|
||||
activationPolicy: 'NEVER',
|
||||
availabilityType: 'ZONAL',
|
||||
tier: 'db-custom-1-3840'
|
||||
}
|
||||
})
|
||||
if (url.hostname === 'certificatemanager.googleapis.com') return Response.json({
|
||||
managed: { domains: ['*.relay-staging.onorca.dev'], state: 'ACTIVE' }
|
||||
})
|
||||
if (url.pathname.includes('/instanceGroupManagers/')) {
|
||||
const name = url.pathname.split('/').at(-1)!
|
||||
return Response.json({
|
||||
name,
|
||||
targetSize: 0,
|
||||
size: '0',
|
||||
instanceGroup: `projects/project/zones/zone/instanceGroups/${name}`,
|
||||
instanceTemplate: `projects/project/global/instanceTemplates/template-${name}`,
|
||||
status: { isStable: true }
|
||||
})
|
||||
}
|
||||
if (url.pathname.includes('/instanceTemplates/')) return Response.json({
|
||||
properties: { metadata: { items: [{
|
||||
key: 'startup-script',
|
||||
value: `SECRET_TEXT\nORCA_RELAY_IMAGE_DIGEST=%s\\n' '${digest}'`
|
||||
}] } }
|
||||
})
|
||||
if (url.pathname.endsWith('/getHealth')) return Response.json([])
|
||||
throw new Error(`Unexpected request to ${url.hostname}${url.pathname}`)
|
||||
}
|
||||
|
||||
const result = await readResourceInventory(
|
||||
RELAY_OPS_ENVIRONMENTS.staging,
|
||||
gcloud,
|
||||
fetchImpl
|
||||
)
|
||||
|
||||
expect(publicProbeCalls).toBe(0)
|
||||
expect(result.cells.every((cell) => cell.targetSize === 0)).toBe(true)
|
||||
expect(result.cells.every((cell) => cell.endpoint.health === null)).toBe(true)
|
||||
expect(result.cells.every((cell) => cell.imageDigest === digest)).toBe(true)
|
||||
expect(JSON.stringify(result)).not.toContain('SECRET_TEXT')
|
||||
})
|
||||
|
||||
it('re-asks a MIG read that failed once before calling a cell powered-unknown', async () => {
|
||||
const parkedCell = RELAY_OPS_ENVIRONMENTS.staging.cells[0]!
|
||||
const waits: number[] = []
|
||||
let parkedMigCalls = 0
|
||||
const result = await readResourceInventory(
|
||||
RELAY_OPS_ENVIRONMENTS.staging,
|
||||
sleepingStagingGcloud,
|
||||
sleepingStagingFetch((migName) => {
|
||||
if (!migName.endsWith(parkedCell.hostname)) return 'ok'
|
||||
parkedMigCalls += 1
|
||||
return parkedMigCalls === 1 ? 'throw' : 'ok'
|
||||
}),
|
||||
{ wait: async (ms) => { waits.push(ms) } }
|
||||
)
|
||||
|
||||
const parked = result.cells.find((cell) => cell.cellId === parkedCell.cellId)!
|
||||
// The MIG was fine and parked at zero; one transient read must not erase that reading.
|
||||
expect(parked.targetSize).toBe(0)
|
||||
expect(parkedMigCalls).toBe(2)
|
||||
expect(waits).toEqual([1_000])
|
||||
expect(result.warnings).toEqual([])
|
||||
})
|
||||
|
||||
it('reports a MIG unavailable only when the retry fails too', async () => {
|
||||
const parkedCell = RELAY_OPS_ENVIRONMENTS.staging.cells[0]!
|
||||
const waits: number[] = []
|
||||
let parkedMigCalls = 0
|
||||
const result = await readResourceInventory(
|
||||
RELAY_OPS_ENVIRONMENTS.staging,
|
||||
sleepingStagingGcloud,
|
||||
sleepingStagingFetch((migName) => {
|
||||
if (!migName.endsWith(parkedCell.hostname)) return 'ok'
|
||||
parkedMigCalls += 1
|
||||
return 'throw'
|
||||
}),
|
||||
{ wait: async (ms) => { waits.push(ms) } }
|
||||
)
|
||||
|
||||
const parked = result.cells.find((cell) => cell.cellId === parkedCell.cellId)!
|
||||
expect(parked.targetSize).toBeNull()
|
||||
expect(parked.backendHealth).toBe('unknown')
|
||||
expect(parkedMigCalls).toBe(2)
|
||||
expect(waits).toEqual([1_000])
|
||||
expect(result.warnings).toEqual([
|
||||
`${parkedCell.hostname.toUpperCase()} MIG inventory is unavailable.`
|
||||
])
|
||||
})
|
||||
|
||||
it('does not re-ask a MIG read the API answered with 404', async () => {
|
||||
const missingCell = RELAY_OPS_ENVIRONMENTS.staging.cells[0]!
|
||||
const waits: number[] = []
|
||||
let missingMigCalls = 0
|
||||
const result = await readResourceInventory(
|
||||
RELAY_OPS_ENVIRONMENTS.staging,
|
||||
sleepingStagingGcloud,
|
||||
sleepingStagingFetch((migName) => {
|
||||
if (!migName.endsWith(missingCell.hostname)) return 'ok'
|
||||
missingMigCalls += 1
|
||||
return 'missing'
|
||||
}),
|
||||
{ wait: async (ms) => { waits.push(ms) } }
|
||||
)
|
||||
|
||||
expect(result.cells.find((cell) => cell.cellId === missingCell.cellId)!.targetSize).toBeNull()
|
||||
expect(missingMigCalls).toBe(1)
|
||||
expect(waits).toEqual([])
|
||||
})
|
||||
|
||||
it('represents missing credentials as unknown inventory, never sleeping', async () => {
|
||||
const gcloud: GcloudClient = {
|
||||
accessToken: async () => { throw new Error('sensitive context') }
|
||||
}
|
||||
let fetchCalls = 0
|
||||
const result = await readResourceInventory(
|
||||
RELAY_OPS_ENVIRONMENTS.production,
|
||||
gcloud,
|
||||
async () => { fetchCalls += 1; return Response.json({}) }
|
||||
)
|
||||
expect(fetchCalls).toBe(0)
|
||||
expect(result.cells.every((cell) => cell.targetSize === null)).toBe(true)
|
||||
expect(result.cells.every((cell) => cell.backendHealth === 'unknown')).toBe(true)
|
||||
expect(JSON.stringify(result)).not.toContain('sensitive context')
|
||||
})
|
||||
})
|
||||
@@ -1,437 +0,0 @@
|
||||
import { z } from 'zod'
|
||||
import type { RelayOpsEnvironment, RelayOpsCellConfig } from './environment-config.js'
|
||||
import type { GcloudClient } from './gcloud-client.js'
|
||||
import { INCIDENT_MONITOR_THRESHOLDS } from './incident-monitor.js'
|
||||
|
||||
const RunServiceSchema = z.object({
|
||||
template: z.object({
|
||||
scaling: z.object({
|
||||
minInstanceCount: z.number().optional(),
|
||||
maxInstanceCount: z.number().optional()
|
||||
}).optional(),
|
||||
containers: z.array(z.object({ image: z.string() })).min(1)
|
||||
}),
|
||||
conditions: z.array(z.object({ state: z.string() })).default([]),
|
||||
latestReadyRevision: z.string().optional()
|
||||
})
|
||||
|
||||
const SqlInstanceSchema = z.object({
|
||||
state: z.string(),
|
||||
databaseVersion: z.string(),
|
||||
settings: z.object({
|
||||
activationPolicy: z.string(),
|
||||
availabilityType: z.string().optional(),
|
||||
tier: z.string()
|
||||
})
|
||||
})
|
||||
|
||||
const MigSchema = z.object({
|
||||
name: z.string(),
|
||||
targetSize: z.number(),
|
||||
size: z.union([z.string(), z.number()]).transform(Number).optional(),
|
||||
instanceGroup: z.string(),
|
||||
instanceTemplate: z.string(),
|
||||
status: z.object({ isStable: z.boolean().default(false) }).default({ isStable: false })
|
||||
})
|
||||
|
||||
const TemplateSchema = z.object({
|
||||
properties: z.object({
|
||||
metadata: z.object({
|
||||
items: z.array(z.object({ key: z.string(), value: z.string().optional() })).default([])
|
||||
}).optional()
|
||||
})
|
||||
})
|
||||
|
||||
const BackendHealthGroupSchema = z.object({
|
||||
healthStatus: z.array(z.object({ healthState: z.string() })).default([])
|
||||
})
|
||||
const BackendHealthSchema = z.union([
|
||||
BackendHealthGroupSchema,
|
||||
z.array(z.object({ status: BackendHealthGroupSchema }))
|
||||
])
|
||||
|
||||
const CertificateSchema = z.object({
|
||||
expireTime: z.string().optional(),
|
||||
managed: z.object({
|
||||
domains: z.array(z.string()).default([]),
|
||||
state: z.string()
|
||||
})
|
||||
})
|
||||
|
||||
export type EndpointHealth = {
|
||||
health: boolean | null
|
||||
ready: boolean | null
|
||||
latencyMs: number | null
|
||||
}
|
||||
|
||||
export type ServiceInventory = {
|
||||
ready: boolean
|
||||
revision: string | null
|
||||
image: string
|
||||
minInstances: number
|
||||
maxInstances: number
|
||||
}
|
||||
|
||||
export type CellInventory = RelayOpsCellConfig & {
|
||||
migName: string
|
||||
targetSize: number | null
|
||||
runningInstances: number | null
|
||||
stable: boolean | null
|
||||
template: string | null
|
||||
imageDigest: string | null
|
||||
backendHealth: 'healthy' | 'unhealthy' | 'empty' | 'unknown'
|
||||
endpoint: EndpointHealth
|
||||
}
|
||||
|
||||
export type ResourceInventory = {
|
||||
director: ServiceInventory | null
|
||||
auth: ServiceInventory | null
|
||||
sql: {
|
||||
state: string
|
||||
activationPolicy: string
|
||||
tier: string
|
||||
availabilityType: string
|
||||
databaseVersion: string
|
||||
} | null
|
||||
certificate: { state: string; domains: string[]; expireTime: string | null } | null
|
||||
directorEndpoint: EndpointHealth
|
||||
authEndpoint: EndpointHealth
|
||||
cells: CellInventory[]
|
||||
warnings: string[]
|
||||
}
|
||||
|
||||
const unavailableEndpoint = (): EndpointHealth => ({ health: null, ready: null, latencyMs: null })
|
||||
const independentEndpointRetryDelayMs = 11_000
|
||||
const transientProbeRetryDelayMs = 1_000
|
||||
const sleep = async (ms: number): Promise<void> =>
|
||||
await new Promise((resolvePromise) => setTimeout(resolvePromise, ms))
|
||||
|
||||
function finalSegment(value: string): string {
|
||||
return value.split('/').at(-1) ?? value
|
||||
}
|
||||
|
||||
function parseService(value: unknown): ServiceInventory {
|
||||
const service = RunServiceSchema.parse(value)
|
||||
return {
|
||||
ready: service.conditions.length > 0 && service.conditions.every(
|
||||
(condition) => condition.state === 'CONDITION_SUCCEEDED'
|
||||
),
|
||||
revision: service.latestReadyRevision ? finalSegment(service.latestReadyRevision) : null,
|
||||
image: service.template.containers[0]!.image,
|
||||
minInstances: service.template.scaling?.minInstanceCount ?? 0,
|
||||
maxInstances: service.template.scaling?.maxInstanceCount ?? 0
|
||||
}
|
||||
}
|
||||
|
||||
class GoogleApiError extends Error {
|
||||
constructor(readonly status: number) {
|
||||
super(`Google API returned ${status}`)
|
||||
}
|
||||
}
|
||||
|
||||
async function googleRequest(
|
||||
fetchImpl: typeof fetch,
|
||||
token: string,
|
||||
url: string,
|
||||
init: RequestInit = {}
|
||||
): Promise<unknown> {
|
||||
const response = await fetchImpl(url, {
|
||||
...init,
|
||||
headers: {
|
||||
authorization: `Bearer ${token}`,
|
||||
...(init.body ? { 'content-type': 'application/json' } : {})
|
||||
},
|
||||
signal: AbortSignal.timeout(30_000)
|
||||
})
|
||||
if (!response.ok) throw new GoogleApiError(response.status)
|
||||
return await response.json()
|
||||
}
|
||||
|
||||
// A 404 is the API's answer about the resource; anything else is the absence of a reading, so re-ask.
|
||||
async function readOnceMore(
|
||||
read: () => Promise<unknown>,
|
||||
wait: (ms: number) => Promise<void>
|
||||
): Promise<unknown> {
|
||||
try {
|
||||
return await read()
|
||||
} catch (error) {
|
||||
if (error instanceof GoogleApiError && error.status === 404) throw error
|
||||
await wait(transientProbeRetryDelayMs)
|
||||
return await read()
|
||||
}
|
||||
}
|
||||
|
||||
// A reading the endpoint actually produced: ok is its answer, latencyMs is that answer's round trip.
|
||||
type PathReading = { ok: boolean; latencyMs: number | null }
|
||||
|
||||
async function probePath(
|
||||
origin: string,
|
||||
path: '/health' | '/ready',
|
||||
fetchImpl: typeof fetch,
|
||||
wait: (ms: number) => Promise<void>
|
||||
): Promise<PathReading> {
|
||||
// null means the request never produced an answer (DNS/TCP/TLS failure or the 8s abort).
|
||||
const attempt = async (): Promise<PathReading | null> => {
|
||||
const startedAt = performance.now()
|
||||
try {
|
||||
const response = await fetchImpl(`${origin}${path}`, {
|
||||
redirect: 'error',
|
||||
signal: AbortSignal.timeout(8_000)
|
||||
})
|
||||
return { ok: response.ok, latencyMs: Math.round(performance.now() - startedAt) }
|
||||
} catch {
|
||||
return null
|
||||
}
|
||||
}
|
||||
const first = await attempt()
|
||||
if (first) return first
|
||||
// A thrown fetch is the absence of a reading, not an unhealthy answer, so re-ask before concluding.
|
||||
await wait(transientProbeRetryDelayMs)
|
||||
return (await attempt()) ?? { ok: false, latencyMs: null }
|
||||
}
|
||||
|
||||
async function endpointProbe(
|
||||
origin: string,
|
||||
fetchImpl: typeof fetch,
|
||||
requiresReady: boolean,
|
||||
wait: (ms: number) => Promise<void>
|
||||
): Promise<EndpointHealth> {
|
||||
const [health, ready] = await Promise.all([
|
||||
probePath(origin, '/health', fetchImpl, wait),
|
||||
requiresReady ? probePath(origin, '/ready', fetchImpl, wait) : null
|
||||
])
|
||||
// Latency is the slowest answering round trip in this probe; retry delays are not serving latency.
|
||||
const latencies = [health.latencyMs, ready?.latencyMs ?? null].filter(
|
||||
(value): value is number => value !== null
|
||||
)
|
||||
return {
|
||||
health: health.ok,
|
||||
ready: ready ? ready.ok : null,
|
||||
latencyMs: latencies.length > 0 ? Math.max(...latencies) : null
|
||||
}
|
||||
}
|
||||
|
||||
export type EndpointProbeOptions = {
|
||||
// Auth serves no /ready by design, so it is judged on /health and latency alone.
|
||||
requiresReady?: boolean
|
||||
wait?: (ms: number) => Promise<void>
|
||||
}
|
||||
|
||||
export async function probeEndpointHealth(
|
||||
origin: string,
|
||||
fetchImpl: typeof fetch,
|
||||
options: EndpointProbeOptions = {}
|
||||
): Promise<EndpointHealth> {
|
||||
const requiresReady = options.requiresReady ?? true
|
||||
const wait = options.wait ?? sleep
|
||||
const accepted = (probe: EndpointHealth): boolean =>
|
||||
probe.health === true &&
|
||||
(!requiresReady || probe.ready === true) &&
|
||||
probe.latencyMs !== null &&
|
||||
probe.latencyMs <= INCIDENT_MONITOR_THRESHOLDS.endpointLatencyMs
|
||||
const first = await endpointProbe(origin, fetchImpl, requiresReady, wait)
|
||||
if (accepted(first)) return first
|
||||
// Outwait Relay's ten-second readiness cache before treating the retry as independent.
|
||||
await wait(independentEndpointRetryDelayMs)
|
||||
return await endpointProbe(origin, fetchImpl, requiresReady, wait)
|
||||
}
|
||||
|
||||
function imageDigest(template: z.infer<typeof TemplateSchema>): string | null {
|
||||
const startupScript = template.properties.metadata?.items.find(
|
||||
(item) => item.key === 'startup-script'
|
||||
)?.value
|
||||
// Return only the immutable digest; startup metadata contains secret names and operational detail.
|
||||
return startupScript?.match(/ORCA_RELAY_IMAGE_DIGEST=%s\\n' '(sha256:[a-f0-9]{64})'/)?.[1] ?? null
|
||||
}
|
||||
|
||||
function backendState(value: unknown): CellInventory['backendHealth'] {
|
||||
const parsedHealth = BackendHealthSchema.parse(value)
|
||||
const groups = Array.isArray(parsedHealth)
|
||||
? parsedHealth.map((group) => group.status)
|
||||
: [parsedHealth]
|
||||
const states = groups.flatMap((group) => group.healthStatus.map((status) => status.healthState))
|
||||
if (states.length === 0) return 'empty'
|
||||
if (states.every((state) => state === 'HEALTHY')) return 'healthy'
|
||||
return 'unhealthy'
|
||||
}
|
||||
|
||||
function unavailableCell(cell: RelayOpsCellConfig, migName: string): CellInventory {
|
||||
return {
|
||||
...cell,
|
||||
migName,
|
||||
targetSize: null,
|
||||
runningInstances: null,
|
||||
stable: null,
|
||||
template: null,
|
||||
imageDigest: null,
|
||||
backendHealth: 'unknown',
|
||||
endpoint: unavailableEndpoint()
|
||||
}
|
||||
}
|
||||
|
||||
async function readCell(
|
||||
environment: RelayOpsEnvironment,
|
||||
cell: RelayOpsCellConfig,
|
||||
mig: z.infer<typeof MigSchema> | null,
|
||||
token: string,
|
||||
fetchImpl: typeof fetch
|
||||
): Promise<CellInventory> {
|
||||
const migName = `${environment.migPrefix}${cell.hostname}`
|
||||
if (!mig) return unavailableCell(cell, migName)
|
||||
// An empty fixed-one MIG cannot serve and must never be woken by observation.
|
||||
const endpoint = mig.targetSize > 0
|
||||
? await probeEndpointHealth(cell.origin, fetchImpl)
|
||||
: unavailableEndpoint()
|
||||
const templateName = finalSegment(mig.instanceTemplate)
|
||||
const [templateResult, healthResult] = await Promise.allSettled([
|
||||
googleRequest(
|
||||
fetchImpl,
|
||||
token,
|
||||
`https://compute.googleapis.com/compute/v1/projects/${environment.project}/global/instanceTemplates/${templateName}`
|
||||
),
|
||||
googleRequest(
|
||||
fetchImpl,
|
||||
token,
|
||||
`https://compute.googleapis.com/compute/v1/projects/${environment.project}/global/backendServices/${migName}/getHealth`,
|
||||
{ method: 'POST', body: JSON.stringify({ group: mig.instanceGroup }) }
|
||||
)
|
||||
])
|
||||
return {
|
||||
...cell,
|
||||
migName,
|
||||
targetSize: mig.targetSize,
|
||||
runningInstances: mig.size ?? (mig.status.isStable ? mig.targetSize : null),
|
||||
stable: mig.status.isStable,
|
||||
template: templateName,
|
||||
imageDigest: templateResult.status === 'fulfilled'
|
||||
? imageDigest(TemplateSchema.parse(templateResult.value))
|
||||
: null,
|
||||
backendHealth: healthResult.status === 'fulfilled'
|
||||
? backendState(healthResult.value)
|
||||
: 'unknown',
|
||||
endpoint
|
||||
}
|
||||
}
|
||||
|
||||
function parsed<S extends z.ZodTypeAny>(
|
||||
result: PromiseSettledResult<unknown>,
|
||||
schema: S,
|
||||
warning: string,
|
||||
warnings: string[]
|
||||
): z.infer<S> | null {
|
||||
if (result.status === 'rejected') {
|
||||
warnings.push(warning)
|
||||
return null
|
||||
}
|
||||
const parsedValue = schema.safeParse(result.value)
|
||||
if (!parsedValue.success) {
|
||||
warnings.push(warning)
|
||||
return null
|
||||
}
|
||||
return parsedValue.data
|
||||
}
|
||||
|
||||
function unavailableInventory(environment: RelayOpsEnvironment, warning: string): ResourceInventory {
|
||||
return {
|
||||
director: null,
|
||||
auth: null,
|
||||
sql: null,
|
||||
certificate: null,
|
||||
directorEndpoint: unavailableEndpoint(),
|
||||
authEndpoint: unavailableEndpoint(),
|
||||
cells: environment.cells.map((cell) =>
|
||||
unavailableCell(cell, `${environment.migPrefix}${cell.hostname}`)
|
||||
),
|
||||
warnings: [warning]
|
||||
}
|
||||
}
|
||||
|
||||
export type ResourceInventoryOptions = {
|
||||
wait?: (ms: number) => Promise<void>
|
||||
}
|
||||
|
||||
export async function readResourceInventory(
|
||||
environment: RelayOpsEnvironment,
|
||||
gcloud: GcloudClient,
|
||||
fetchImpl: typeof fetch = fetch,
|
||||
options: ResourceInventoryOptions = {}
|
||||
): Promise<ResourceInventory> {
|
||||
const wait = options.wait ?? sleep
|
||||
let token: string
|
||||
try {
|
||||
token = await gcloud.accessToken()
|
||||
} catch {
|
||||
return unavailableInventory(
|
||||
environment,
|
||||
'Google Cloud credentials are unavailable. Run gcloud auth login.'
|
||||
)
|
||||
}
|
||||
const runUrl = (service: string) =>
|
||||
`https://run.googleapis.com/v2/projects/${environment.project}/locations/${environment.region}/services/${service}`
|
||||
const migUrl = (cell: RelayOpsCellConfig) =>
|
||||
`https://compute.googleapis.com/compute/v1/projects/${environment.project}/zones/${cell.zone}/instanceGroupManagers/${environment.migPrefix}${cell.hostname}`
|
||||
const settled = await Promise.allSettled([
|
||||
googleRequest(fetchImpl, token, runUrl(environment.directorService)),
|
||||
googleRequest(fetchImpl, token, runUrl(environment.authService)),
|
||||
googleRequest(
|
||||
fetchImpl,
|
||||
token,
|
||||
`https://sqladmin.googleapis.com/sql/v1beta4/projects/${environment.project}/instances/${environment.sqlInstance}`
|
||||
),
|
||||
googleRequest(
|
||||
fetchImpl,
|
||||
token,
|
||||
`https://certificatemanager.googleapis.com/v1/projects/${environment.project}/locations/global/certificates/${environment.certificateName}`
|
||||
),
|
||||
// One transient Compute read must never become a verdict on a cell's power state.
|
||||
...environment.cells.map((cell) =>
|
||||
readOnceMore(async () => await googleRequest(fetchImpl, token, migUrl(cell)), wait)
|
||||
)
|
||||
])
|
||||
const warnings: string[] = []
|
||||
const directorValue = parsed(settled[0]!, RunServiceSchema, 'Director service inventory is unavailable.', warnings)
|
||||
const authValue = parsed(settled[1]!, RunServiceSchema, 'Auth service inventory is unavailable.', warnings)
|
||||
const sqlValue = parsed(settled[2]!, SqlInstanceSchema, 'Cloud SQL inventory is unavailable.', warnings)
|
||||
const certificateValue = parsed(
|
||||
settled[3]!, CertificateSchema, 'TLS certificate inventory is unavailable.', warnings
|
||||
)
|
||||
const migValues = environment.cells.map((cell, index) => parsed(
|
||||
settled[index + 4]!,
|
||||
MigSchema,
|
||||
`${cell.hostname.toUpperCase()} MIG inventory is unavailable.`,
|
||||
warnings
|
||||
))
|
||||
const controlPlaneSleeping =
|
||||
environment.id === 'staging' && sqlValue?.settings.activationPolicy === 'NEVER'
|
||||
// Health probes would cold-start scale-to-zero Cloud Run services, so sleeping staging is inventory-only.
|
||||
const [directorEndpoint, authEndpoint] = controlPlaneSleeping
|
||||
? [unavailableEndpoint(), unavailableEndpoint()]
|
||||
: await Promise.all([
|
||||
probeEndpointHealth(environment.directorOrigin, fetchImpl),
|
||||
// The auth service exposes no /ready, so requiring it would fail every first probe.
|
||||
probeEndpointHealth(environment.authOrigin, fetchImpl, { requiresReady: false })
|
||||
])
|
||||
const cells = await Promise.all(environment.cells.map((cell, index) =>
|
||||
readCell(environment, cell, migValues[index] ?? null, token, fetchImpl)
|
||||
))
|
||||
return {
|
||||
director: directorValue ? parseService(directorValue) : null,
|
||||
auth: authValue ? parseService(authValue) : null,
|
||||
sql: sqlValue ? {
|
||||
state: sqlValue.state,
|
||||
activationPolicy: sqlValue.settings.activationPolicy,
|
||||
tier: sqlValue.settings.tier,
|
||||
availabilityType: sqlValue.settings.availabilityType ?? 'unknown',
|
||||
databaseVersion: sqlValue.databaseVersion
|
||||
} : null,
|
||||
certificate: certificateValue ? {
|
||||
state: certificateValue.managed.state,
|
||||
domains: certificateValue.managed.domains,
|
||||
expireTime: certificateValue.expireTime ?? null
|
||||
} : null,
|
||||
directorEndpoint,
|
||||
authEndpoint,
|
||||
cells,
|
||||
warnings
|
||||
}
|
||||
}
|
||||
@@ -1,19 +0,0 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { parseStagingPowerRequest } from './staging-workflow.js'
|
||||
|
||||
describe('parseStagingPowerRequest', () => {
|
||||
it('accepts the exact reviewed confirmations', () => {
|
||||
expect(parseStagingPowerRequest({ mode: 'status', confirmation: '' })).toEqual({
|
||||
mode: 'status', confirmation: ''
|
||||
})
|
||||
expect(parseStagingPowerRequest({ mode: 'wake', confirmation: 'WAKE_STAGING' }).mode).toBe('wake')
|
||||
expect(parseStagingPowerRequest({ mode: 'sleep', confirmation: 'SLEEP_STAGING' }).mode).toBe('sleep')
|
||||
})
|
||||
|
||||
it('rejects missing, swapped, or additional fields', () => {
|
||||
expect(() => parseStagingPowerRequest({ mode: 'wake', confirmation: '' })).toThrow()
|
||||
expect(() => parseStagingPowerRequest({ mode: 'sleep', confirmation: 'WAKE_STAGING' })).toThrow()
|
||||
expect(() => parseStagingPowerRequest({ mode: 'production', confirmation: '' })).toThrow()
|
||||
expect(() => parseStagingPowerRequest({ mode: 'status', confirmation: '', project: 'other' })).toThrow()
|
||||
})
|
||||
})
|
||||
@@ -1,36 +0,0 @@
|
||||
import { execFile } from 'node:child_process'
|
||||
import { promisify } from 'node:util'
|
||||
import { z } from 'zod'
|
||||
import { RELAY_GITHUB_REPOSITORY, relayWorkflowFile } from './relay-repository.js'
|
||||
|
||||
const execFileAsync = promisify(execFile)
|
||||
const DispatchSchema = z.discriminatedUnion('mode', [
|
||||
z.object({ mode: z.literal('status'), confirmation: z.literal('') }).strict(),
|
||||
z.object({ mode: z.literal('wake'), confirmation: z.literal('WAKE_STAGING') }).strict(),
|
||||
z.object({ mode: z.literal('sleep'), confirmation: z.literal('SLEEP_STAGING') }).strict()
|
||||
])
|
||||
|
||||
export type StagingPowerRequest = z.infer<typeof DispatchSchema>
|
||||
|
||||
export function parseStagingPowerRequest(value: unknown): StagingPowerRequest {
|
||||
return DispatchSchema.parse(value)
|
||||
}
|
||||
|
||||
export async function dispatchStagingPowerWorkflow(request: StagingPowerRequest): Promise<void> {
|
||||
const args = [
|
||||
'workflow', 'run', relayWorkflowFile('power-relay-staging.yml'),
|
||||
'--repo', RELAY_GITHUB_REPOSITORY,
|
||||
'-f', `mode=${request.mode}`,
|
||||
'-f', 'wake-cells=configured'
|
||||
]
|
||||
if (request.confirmation) args.push('-f', `confirmation=${request.confirmation}`)
|
||||
try {
|
||||
await execFileAsync('gh', args, {
|
||||
encoding: 'utf8',
|
||||
timeout: 30_000,
|
||||
maxBuffer: 1024 * 1024
|
||||
})
|
||||
} catch {
|
||||
throw new Error('Staging power workflow dispatch failed')
|
||||
}
|
||||
}
|
||||
@@ -1,4 +0,0 @@
|
||||
{
|
||||
"extends": "./tsconfig.json",
|
||||
"exclude": ["src/**/*.test.ts"]
|
||||
}
|
||||
@@ -1,11 +0,0 @@
|
||||
{
|
||||
"extends": "../../tsconfig.base.json",
|
||||
"compilerOptions": {
|
||||
"rootDir": "src",
|
||||
"outDir": "dist",
|
||||
"types": ["node", "vitest"],
|
||||
"exactOptionalPropertyTypes": true,
|
||||
"verbatimModuleSyntax": true
|
||||
},
|
||||
"include": ["src/**/*.ts"]
|
||||
}
|
||||
@@ -1,75 +0,0 @@
|
||||
import { readFileSync } from 'node:fs'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
import {
|
||||
isRelayCellConnectionHardCap,
|
||||
RELAY_ADMISSION_BUDGETS,
|
||||
RELAY_CELL_ADMISSION_BOUNDS,
|
||||
RELAY_CELL_CONNECTION_HARD_CAP,
|
||||
RELAY_CELL_CONNECTION_HARD_CAPS,
|
||||
relayCellAdmissionBounds
|
||||
} from '@orca-cloud/relay-contract'
|
||||
import { describe, expect, it } from 'vitest'
|
||||
|
||||
// Why: dev scripts run standalone in CI and Terraform cannot read TypeScript, so neither can
|
||||
// import the constant. Both restate it instead. This asserts every restatement still agrees, so
|
||||
// raising the cap fails loudly here rather than silently leaving a surface behind.
|
||||
// Repository root, and every path read below stays inside the Relay tree, so this survives the
|
||||
// move under cloud/ in the public repository.
|
||||
const repositoryRoot = fileURLToPath(new URL('../../../', import.meta.url))
|
||||
const read = (path: string): string => readFileSync(`${repositoryRoot}${path}`, 'utf8')
|
||||
|
||||
describe('cell connection hard cap stays consistent across surfaces that cannot import it', () => {
|
||||
it('derives its own bounds from the cap', () => {
|
||||
expect(RELAY_CELL_ADMISSION_BOUNDS.hardCap).toBe(RELAY_CELL_CONNECTION_HARD_CAP)
|
||||
expect(RELAY_CELL_ADMISSION_BOUNDS.socketAdmissionCeiling).toBe(
|
||||
RELAY_CELL_CONNECTION_HARD_CAP - RELAY_ADMISSION_BUDGETS.reservedHostControls
|
||||
)
|
||||
expect(RELAY_CELL_ADMISSION_BOUNDS.maxUnobservedBound).toBe(
|
||||
RELAY_CELL_ADMISSION_BOUNDS.socketAdmissionCeiling - 1
|
||||
)
|
||||
for (const hardCap of RELAY_CELL_CONNECTION_HARD_CAPS) {
|
||||
const bounds = relayCellAdmissionBounds(hardCap)
|
||||
expect(bounds.socketAdmissionCeiling).toBe(
|
||||
hardCap - RELAY_ADMISSION_BUDGETS.reservedHostControls
|
||||
)
|
||||
expect(bounds.maxUnobservedBound).toBe(bounds.socketAdmissionCeiling - 1)
|
||||
}
|
||||
})
|
||||
|
||||
it.each([
|
||||
['dev/scripts/relay-recovery-wave-gate.mjs', /targetConnectionCap:\s*(\d+)/],
|
||||
['dev/scripts/deploy-relay-gce-multi-target.mjs', /DEFAULT_CONNECTION_CEILING\s*=\s*(\d+)/],
|
||||
['dev/scripts/deploy-relay-gce-multi-target.mjs', /CUTOVER_CONNECTION_HARD_CAP\s*=\s*(\d+)/]
|
||||
])('%s matches the contract cap', (path, pattern) => {
|
||||
const match = read(path).match(pattern)
|
||||
expect(match, `${path} no longer declares ${pattern}`).not.toBeNull()
|
||||
expect(Number(match![1])).toBe(RELAY_CELL_CONNECTION_HARD_CAP)
|
||||
})
|
||||
|
||||
it('every production cell declaring a cap uses a supported contract cap', () => {
|
||||
const declared = [
|
||||
...read('infra/terraform/environments/production.tfvars').matchAll(
|
||||
/connection_hard_cap\s*=\s*(\d+)/g
|
||||
)
|
||||
].map((match) => Number(match[1]))
|
||||
expect(declared.length).toBeGreaterThan(0)
|
||||
expect(new Set(declared).has(RELAY_CELL_CONNECTION_HARD_CAP)).toBe(true)
|
||||
expect(declared.every((hardCap) => isRelayCellConnectionHardCap(hardCap))).toBe(true)
|
||||
})
|
||||
|
||||
it('the Terraform cell validation accepts exactly the supported contract caps', () => {
|
||||
const match = read('infra/terraform/variables.tf').match(
|
||||
/contains\(\[([^\]]+)\], cell\.connection_hard_cap\)/
|
||||
)
|
||||
expect(match, 'variables.tf no longer validates connection_hard_cap').not.toBeNull()
|
||||
expect(match![1]!.split(',').map((value) => Number(value.trim()))).toEqual([
|
||||
...RELAY_CELL_CONNECTION_HARD_CAPS
|
||||
])
|
||||
})
|
||||
|
||||
it('the Terraform unobserved bound leaves the contract rebind reserve', () => {
|
||||
expect(read('infra/terraform/variables.tf')).toContain(
|
||||
'cell.connection_unobserved_bound < cell.connection_hard_cap - 100'
|
||||
)
|
||||
})
|
||||
})
|
||||
@@ -1,55 +0,0 @@
|
||||
import { readFileSync } from 'node:fs'
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { loadRelayConfig } from './config.js'
|
||||
|
||||
// A same-cap roll swaps only the image: the cell keeps the env its startup script already
|
||||
// wrote. So every image must boot on exactly the variables that script renders, and a
|
||||
// config change that needs a new variable is a template change, not an image-only roll.
|
||||
const template = readFileSync(
|
||||
new URL('../../../infra/terraform/relay-gce-startup.sh.tftpl', import.meta.url),
|
||||
'utf8'
|
||||
)
|
||||
|
||||
// One plausible production value per variable the template can write.
|
||||
const RENDERED: Record<string, string> = {
|
||||
DATABASE_URL: 'postgres://relay@127.0.0.1:5432/orca_relay',
|
||||
ORCA_RELAY_ASSIGNMENT_SIGNING_KEY: 'assignment-key-with-at-least-thirty-two-bytes',
|
||||
ORCA_RELAY_PUBLIC_URL: 'https://c25.relay.onorca.dev',
|
||||
ORCA_RELAY_CELL_URL: 'https://c25.relay.onorca.dev',
|
||||
ORCA_RELAY_AUTH_ISSUER: 'https://auth.onorca.dev',
|
||||
ORCA_RELAY_AUTH_AUDIENCE: 'orca-relay',
|
||||
ORCA_RELAY_JWKS_URL: 'https://auth.onorca.dev/.well-known/jwks.json',
|
||||
ORCA_RELAY_ROLE: 'cell',
|
||||
ORCA_RELAY_CELL_ID: 'production-gce-c25',
|
||||
ORCA_RELAY_REGION: 'asia-east2',
|
||||
ORCA_RELAY_CELL_CAPACITY: '3000',
|
||||
ORCA_RELAY_DATABASE_POOL_MAX: '16',
|
||||
ORCA_RELAY_CELL_CONNECTION_HARD_CAP: '3000',
|
||||
ORCA_RELAY_CELL_CONNECTION_UNOBSERVED_BOUND: '60',
|
||||
ORCA_RELAY_CELLS_JSON: '[]',
|
||||
ORCA_RELAY_ADMIN_AUDIENCE: 'https://relay.onorca.dev/v1/admin/drain',
|
||||
ORCA_RELAY_DEPLOY_SERVICE_ACCOUNT: 'deploy@example.iam.gserviceaccount.com',
|
||||
ORCA_RELAY_CAPACITY_SERVICE_ACCOUNT: 'capacity@example.iam.gserviceaccount.com',
|
||||
ORCA_RELAY_ASIA_PROOF_SERVICE_ACCOUNT: 'asia-proof@example.iam.gserviceaccount.com',
|
||||
ORCA_RELAY_RUNTIME_SERVICE_ACCOUNT: 'relay-cell@example.iam.gserviceaccount.com',
|
||||
ORCA_RELAY_REHOME_DIRECTOR_SERVICE_ACCOUNT: 'relay-director@example.iam.gserviceaccount.com',
|
||||
ORCA_RELAY_REHOME_AUDIENCE: 'https://relay.onorca.dev/v1/admin/host-drain',
|
||||
ORCA_RELAY_DIRECTOR_URL: 'https://relay.onorca.dev',
|
||||
ORCA_RELAY_HEARTBEAT_AUDIENCE: 'https://relay.onorca.dev/v1/admin/cell-heartbeat',
|
||||
ORCA_RELAY_IMAGE_DIGEST: `sha256:${'a'.repeat(64)}`
|
||||
}
|
||||
|
||||
const TEMPLATE_VARIABLES = [...template.matchAll(/printf '([A-Z0-9_]+)=/g)].map(([, name]) => name!)
|
||||
|
||||
describe('cell startup environment', () => {
|
||||
it('boots a cell on exactly the variables the startup template writes', () => {
|
||||
expect(TEMPLATE_VARIABLES.length).toBeGreaterThan(20)
|
||||
expect(TEMPLATE_VARIABLES.filter((name) => RENDERED[name] === undefined)).toEqual([])
|
||||
const env = Object.fromEntries(TEMPLATE_VARIABLES.map((name) => [name, RENDERED[name]]))
|
||||
expect(loadRelayConfig(env)).toMatchObject({
|
||||
role: 'cell',
|
||||
cellId: 'production-gce-c25',
|
||||
databaseUrl: RENDERED.DATABASE_URL
|
||||
})
|
||||
})
|
||||
})
|
||||
@@ -1,163 +0,0 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import {
|
||||
formatAssignmentInventorySnapshot,
|
||||
type AssignmentInventorySnapshot
|
||||
} from './assignment-inventory-snapshot.js'
|
||||
|
||||
// The enable workflow reads the director's rehome inventory line out of Cloud
|
||||
// Logging with a parser that lives in another language, in another package, and
|
||||
// is never exercised against the formatter that writes the line. When
|
||||
// `hostNotArrivedLast24Hours` shipped, the parser read a healthy line as no
|
||||
// evidence at all and the run failed closed, disabling the durable switch. This
|
||||
// test is the missing edge: the real formatter's output, through the real
|
||||
// parser, so a future field fails here instead of in an operator's run.
|
||||
|
||||
type InventoryEvidence = {
|
||||
active: number
|
||||
awaitingReceipt: number
|
||||
targetRegistered: number
|
||||
completedLast24Hours: number
|
||||
abortedLast24Hours: number
|
||||
hostNotArrivedLast24Hours: number | null
|
||||
oldestActiveAgeMs: number | null
|
||||
}
|
||||
|
||||
// Imported through a computed URL on purpose: the script is plain ESM outside
|
||||
// this package's compile scope, so a static import would not resolve.
|
||||
async function loadParser(): Promise<(entries: unknown[], options: unknown) => InventoryEvidence> {
|
||||
const source = new URL(
|
||||
'../../../dev/scripts/relay-rehome-aggregate-evidence.mjs',
|
||||
import.meta.url
|
||||
).href
|
||||
const loaded: unknown = await import(/* @vite-ignore */ source)
|
||||
if (!(loaded !== null && typeof loaded === 'object' && 'parseRegionalRehomeInventory' in loaded)) {
|
||||
throw new Error('relay-rehome-aggregate-evidence.mjs no longer exports its parser')
|
||||
}
|
||||
const parse = loaded.parseRegionalRehomeInventory
|
||||
if (typeof parse !== 'function') throw new Error('parseRegionalRehomeInventory is not callable')
|
||||
return (entries, options) => readEvidence(parse(entries, options))
|
||||
}
|
||||
|
||||
function readEvidence(value: unknown): InventoryEvidence {
|
||||
if (value === null || typeof value !== 'object') throw new Error('parser returned no evidence')
|
||||
const counts = ['active', 'awaitingReceipt', 'targetRegistered', 'completedLast24Hours', 'abortedLast24Hours'] as const
|
||||
const evidence: Record<string, number | null> = {}
|
||||
for (const key of [...counts, 'hostNotArrivedLast24Hours', 'oldestActiveAgeMs'] as const) {
|
||||
if (!(key in value)) throw new Error(`parser dropped ${key}`)
|
||||
const read: unknown = Reflect.get(value, key)
|
||||
if (read !== null && typeof read !== 'number') throw new Error(`${key} is not a count`)
|
||||
evidence[key] = read
|
||||
}
|
||||
for (const key of counts) {
|
||||
if (evidence[key] === null) throw new Error(`${key} must be a number`)
|
||||
}
|
||||
return {
|
||||
active: Number(evidence['active']),
|
||||
awaitingReceipt: Number(evidence['awaitingReceipt']),
|
||||
targetRegistered: Number(evidence['targetRegistered']),
|
||||
completedLast24Hours: Number(evidence['completedLast24Hours']),
|
||||
abortedLast24Hours: Number(evidence['abortedLast24Hours']),
|
||||
hostNotArrivedLast24Hours: evidence['hostNotArrivedLast24Hours'] ?? null,
|
||||
oldestActiveAgeMs: evidence['oldestActiveAgeMs'] ?? null
|
||||
}
|
||||
}
|
||||
|
||||
function snapshot(
|
||||
regionalRehomes: AssignmentInventorySnapshot['regionalRehomes']
|
||||
): AssignmentInventorySnapshot {
|
||||
return {
|
||||
cells: [],
|
||||
activityLeases: { total: 0, expired: 0, requestUnits: 0 },
|
||||
connectionReservations: { outstanding: 0, lateArrivalDebt: 0 },
|
||||
regionalRehomes
|
||||
}
|
||||
}
|
||||
|
||||
function inventoryLine(snapshotValue: AssignmentInventorySnapshot): string {
|
||||
const line = formatAssignmentInventorySnapshot(snapshotValue).find((candidate) =>
|
||||
candidate.startsWith('[orca-relay] regional rehome inventory ')
|
||||
)
|
||||
if (!line) throw new Error('the formatter no longer emits a rehome inventory line')
|
||||
return line
|
||||
}
|
||||
|
||||
describe('regional rehome inventory line census', () => {
|
||||
it('parses what the director actually prints, field for field', async () => {
|
||||
const parse = await loadParser()
|
||||
const regionalRehomes = {
|
||||
active: 3,
|
||||
awaitingReceipt: 1,
|
||||
targetRegistered: 2,
|
||||
completedLast24Hours: 41,
|
||||
abortedLast24Hours: 12,
|
||||
hostNotArrivedLast24Hours: 5,
|
||||
oldestActiveAgeMs: 77_731_209
|
||||
}
|
||||
const now = Date.parse('2026-09-20T12:00:00Z')
|
||||
|
||||
const evidence = parse(
|
||||
[{ timestamp: '2026-09-20T11:59:00Z', textPayload: inventoryLine(snapshot(regionalRehomes)) }],
|
||||
{ now, maxAgeMs: 5 * 60_000 }
|
||||
)
|
||||
|
||||
expect(evidence).toEqual({ ...regionalRehomes })
|
||||
})
|
||||
|
||||
it('parses the line an idle fleet prints, with no oldest active age', async () => {
|
||||
const parse = await loadParser()
|
||||
const now = Date.parse('2026-09-20T12:00:00Z')
|
||||
|
||||
const evidence = parse(
|
||||
[
|
||||
{
|
||||
timestamp: '2026-09-20T11:59:00Z',
|
||||
textPayload: inventoryLine(
|
||||
snapshot({
|
||||
active: 0,
|
||||
awaitingReceipt: 0,
|
||||
targetRegistered: 0,
|
||||
completedLast24Hours: 0,
|
||||
abortedLast24Hours: 0,
|
||||
hostNotArrivedLast24Hours: 0,
|
||||
oldestActiveAgeMs: null
|
||||
})
|
||||
)
|
||||
}
|
||||
],
|
||||
{ now, maxAgeMs: 5 * 60_000 }
|
||||
)
|
||||
|
||||
expect(evidence.oldestActiveAgeMs).toBeNull()
|
||||
expect(evidence.hostNotArrivedLast24Hours).toBe(0)
|
||||
})
|
||||
|
||||
it('covers every counter the formatter puts on the line', async () => {
|
||||
const parse = await loadParser()
|
||||
// A field the parser ignores is a field the operator never sees, so the
|
||||
// census fails when the formatter gains one and this test is not updated.
|
||||
const line = inventoryLine(
|
||||
snapshot({
|
||||
active: 1,
|
||||
awaitingReceipt: 1,
|
||||
targetRegistered: 1,
|
||||
completedLast24Hours: 1,
|
||||
abortedLast24Hours: 1,
|
||||
hostNotArrivedLast24Hours: 1,
|
||||
oldestActiveAgeMs: 1
|
||||
})
|
||||
)
|
||||
const printed = line
|
||||
.slice('[orca-relay] regional rehome inventory '.length)
|
||||
.split(' ')
|
||||
.map((field) => field.split('=')[0])
|
||||
|
||||
const surfaced = Object.keys(
|
||||
parse([{ timestamp: '2026-09-20T11:59:00Z', textPayload: line }], {
|
||||
now: Date.parse('2026-09-20T12:00:00Z'),
|
||||
maxAgeMs: 5 * 60_000
|
||||
})
|
||||
)
|
||||
|
||||
expect([...printed].sort()).toEqual([...surfaced].sort())
|
||||
})
|
||||
})
|
||||
@@ -1,15 +0,0 @@
|
||||
{
|
||||
"comment": "Production Cloud SQL consumers owned by the private orca-cloud application tree (auth and API services). The relay ships without them, so the values the connection budget needs are published here; the private repository binds every field back to its source in its own CI.",
|
||||
"authInstances": 2,
|
||||
"authPoolMax": 10,
|
||||
"apiInstances": 10,
|
||||
"apiPoolMax": 5,
|
||||
"maxConnections": 500,
|
||||
"sources": {
|
||||
"authInstances": "private apps tfvars: auth service max instances",
|
||||
"authPoolMax": "private auth service: pg.Pool max",
|
||||
"apiInstances": "private apps tfvars: API service max instances",
|
||||
"apiPoolMax": "private API service: pg.Pool max",
|
||||
"maxConnections": "measured SHOW max_connections = 500 on the live instance 2026-09-16; no max_connections flag is set, so this is the tier default and the previous 400 was an unverified assumption about it"
|
||||
}
|
||||
}
|
||||
@@ -1,151 +0,0 @@
|
||||
{
|
||||
"comment": "Production 2026-10-21 07:44-07:58Z, read-only: Cloud Run director 503s per minute (request_count, code 503) and every director runtime-metrics line, instance ids replaced. d5 stalled on its database 07:55-07:57.",
|
||||
"fleet503PerMinute": {"2026-10-21T07:44":6,"2026-10-21T07:45":7,"2026-10-21T07:46":5,"2026-10-21T07:47":55,"2026-10-21T07:48":11,"2026-10-21T07:49":10,"2026-10-21T07:50":12,"2026-10-21T07:51":4,"2026-10-21T07:52":6,"2026-10-21T07:53":8,"2026-10-21T07:54":6,"2026-10-21T07:55":57,"2026-10-21T07:56":173,"2026-10-21T07:57":45},
|
||||
"directorMetricsLines": [
|
||||
{"timestamp":"2026-10-21T07:44:05.179655Z","instance":"d1","stickyRejectionsByReasonDelta":{"host-rate-limited":1},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":1}},
|
||||
{"timestamp":"2026-10-21T07:44:08.062180Z","instance":"d2","stickyRejectionsByReasonDelta":{"host-in-flight":1,"host-rate-limited":2},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":3}},
|
||||
{"timestamp":"2026-10-21T07:44:13.822472Z","instance":"d3","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"relay_assignment_row_busy":1}},
|
||||
{"timestamp":"2026-10-21T07:44:14.159472Z","instance":"d4","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:44:25.765825Z","instance":"d5","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:44:35.181579Z","instance":"d1","stickyRejectionsByReasonDelta":{"host-rate-limited":1},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":1}},
|
||||
{"timestamp":"2026-10-21T07:44:38.060162Z","instance":"d2","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:44:43.823368Z","instance":"d3","stickyRejectionsByReasonDelta":{"host-rate-limited":1},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":1}},
|
||||
{"timestamp":"2026-10-21T07:44:44.158999Z","instance":"d4","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:44:55.766172Z","instance":"d5","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:45:05.180654Z","instance":"d1","stickyRejectionsByReasonDelta":{"host-rate-limited":2},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":2}},
|
||||
{"timestamp":"2026-10-21T07:45:08.060467Z","instance":"d2","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:45:13.824498Z","instance":"d3","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:45:14.159451Z","instance":"d4","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:45:25.764958Z","instance":"d5","stickyRejectionsByReasonDelta":{"host-rate-limited":1},"placementRejectionsByReasonDelta":{"host-rate-limited":1},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":1}},
|
||||
{"timestamp":"2026-10-21T07:45:35.181560Z","instance":"d1","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:45:38.062560Z","instance":"d2","stickyRejectionsByReasonDelta":{"host-rate-limited":1},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":1}},
|
||||
{"timestamp":"2026-10-21T07:45:43.831975Z","instance":"d3","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:45:44.160118Z","instance":"d4","placementRejectionsByReasonDelta":{"host-rate-limited":1},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:45:55.767508Z","instance":"d5","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:46:05.181410Z","instance":"d1","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:46:08.061944Z","instance":"d2","stickyRejectionsByReasonDelta":{"host-rate-limited":3},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":3}},
|
||||
{"timestamp":"2026-10-21T07:46:13.824904Z","instance":"d3","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:46:14.160287Z","instance":"d4","stickyRejectionsByReasonDelta":{"host-rate-limited":1},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":1}},
|
||||
{"timestamp":"2026-10-21T07:46:25.766494Z","instance":"d5","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:46:35.180987Z","instance":"d1","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:46:38.061343Z","instance":"d2","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:46:43.824785Z","instance":"d3","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:46:44.161722Z","instance":"d4","stickyRejectionsByReasonDelta":{"host-rate-limited":1},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":1}},
|
||||
{"timestamp":"2026-10-21T07:46:55.768452Z","instance":"d5","stickyRejectionsByReasonDelta":{"host-rate-limited":1},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":1}},
|
||||
{"timestamp":"2026-10-21T07:47:05.182855Z","instance":"d1","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:47:08.066053Z","instance":"d2","stickyRejectionsByReasonDelta":{"host-rate-limited":2},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":2}},
|
||||
{"timestamp":"2026-10-21T07:47:13.825495Z","instance":"d3","stickyRejectionsByReasonDelta":{"host-rate-limited":3},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":3}},
|
||||
{"timestamp":"2026-10-21T07:47:14.161928Z","instance":"d4","stickyRejectionsByReasonDelta":{"host-rate-limited":1},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"relay_assignment_row_busy":1,"sticky-lane":1}},
|
||||
{"timestamp":"2026-10-21T07:47:25.767488Z","instance":"d5","stickyRejectionsByReasonDelta":{"host-rate-limited":4},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":4}},
|
||||
{"timestamp":"2026-10-21T07:47:35.183676Z","instance":"d1","stickyRejectionsByReasonDelta":{"host-rate-limited":2},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":2}},
|
||||
{"timestamp":"2026-10-21T07:47:38.064515Z","instance":"d2","stickyRejectionsByReasonDelta":{"host-rate-limited":2},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":2}},
|
||||
{"timestamp":"2026-10-21T07:47:43.827753Z","instance":"d3","stickyRejectionsByReasonDelta":{"host-in-flight":1,"host-rate-limited":1,"wait-timeout":9},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":11}},
|
||||
{"timestamp":"2026-10-21T07:47:44.161502Z","instance":"d4","stickyRejectionsByReasonDelta":{"host-in-flight":1,"wait-timeout":7},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":8}},
|
||||
{"timestamp":"2026-10-21T07:47:55.769396Z","instance":"d5","stickyRejectionsByReasonDelta":{"wait-timeout":9},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":9}},
|
||||
{"timestamp":"2026-10-21T07:48:05.184152Z","instance":"d1","stickyRejectionsByReasonDelta":{"host-rate-limited":1,"wait-timeout":10},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":11}},
|
||||
{"timestamp":"2026-10-21T07:48:08.069122Z","instance":"d2","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:48:13.826317Z","instance":"d3","stickyRejectionsByReasonDelta":{"wait-timeout":1},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":1}},
|
||||
{"timestamp":"2026-10-21T07:48:14.162388Z","instance":"d4","stickyRejectionsByReasonDelta":{"host-rate-limited":1,"wait-timeout":1},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":2}},
|
||||
{"timestamp":"2026-10-21T07:48:25.770092Z","instance":"d5","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:48:35.183997Z","instance":"d1","stickyRejectionsByReasonDelta":{"host-rate-limited":1},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":1}},
|
||||
{"timestamp":"2026-10-21T07:48:38.072360Z","instance":"d2","stickyRejectionsByReasonDelta":{"host-rate-limited":2},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":2}},
|
||||
{"timestamp":"2026-10-21T07:48:43.826028Z","instance":"d3","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:48:44.162106Z","instance":"d4","stickyRejectionsByReasonDelta":{"host-rate-limited":2},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":2}},
|
||||
{"timestamp":"2026-10-21T07:48:55.769497Z","instance":"d5","stickyRejectionsByReasonDelta":{"host-rate-limited":2},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":2}},
|
||||
{"timestamp":"2026-10-21T07:49:05.183794Z","instance":"d1","stickyRejectionsByReasonDelta":{"host-rate-limited":1},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":1}},
|
||||
{"timestamp":"2026-10-21T07:49:08.070369Z","instance":"d2","stickyRejectionsByReasonDelta":{"host-rate-limited":1},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":1}},
|
||||
{"timestamp":"2026-10-21T07:49:13.826477Z","instance":"d3","stickyRejectionsByReasonDelta":{"host-rate-limited":1},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":1}},
|
||||
{"timestamp":"2026-10-21T07:49:14.162921Z","instance":"d4","stickyRejectionsByReasonDelta":{"host-rate-limited":2},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"relay_assignment_row_busy":1,"sticky-lane":2}},
|
||||
{"timestamp":"2026-10-21T07:49:25.771495Z","instance":"d5","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:49:35.183472Z","instance":"d1","stickyRejectionsByReasonDelta":{"host-rate-limited":2},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":2}},
|
||||
{"timestamp":"2026-10-21T07:49:38.072836Z","instance":"d2","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:49:43.826252Z","instance":"d3","stickyRejectionsByReasonDelta":{"host-rate-limited":2},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":2}},
|
||||
{"timestamp":"2026-10-21T07:49:44.163115Z","instance":"d4","stickyRejectionsByReasonDelta":{"host-rate-limited":1},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":1}},
|
||||
{"timestamp":"2026-10-21T07:49:55.769846Z","instance":"d5","stickyRejectionsByReasonDelta":{"host-rate-limited":1},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":1}},
|
||||
{"timestamp":"2026-10-21T07:50:05.184917Z","instance":"d1","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:50:08.072488Z","instance":"d2","stickyRejectionsByReasonDelta":{"host-rate-limited":1},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":1}},
|
||||
{"timestamp":"2026-10-21T07:50:13.827130Z","instance":"d3","stickyRejectionsByReasonDelta":{"host-rate-limited":3},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":3}},
|
||||
{"timestamp":"2026-10-21T07:50:14.163400Z","instance":"d4","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:50:25.769945Z","instance":"d5","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:50:35.184832Z","instance":"d1","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:50:38.075017Z","instance":"d2","stickyRejectionsByReasonDelta":{"host-rate-limited":2},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":2}},
|
||||
{"timestamp":"2026-10-21T07:50:43.827931Z","instance":"d3","stickyRejectionsByReasonDelta":{"host-rate-limited":1},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":1}},
|
||||
{"timestamp":"2026-10-21T07:50:44.164604Z","instance":"d4","stickyRejectionsByReasonDelta":{"host-rate-limited":1},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":1}},
|
||||
{"timestamp":"2026-10-21T07:50:55.772344Z","instance":"d5","stickyRejectionsByReasonDelta":{"host-rate-limited":3},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":3}},
|
||||
{"timestamp":"2026-10-21T07:51:05.184087Z","instance":"d1","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"relay_assignment_row_busy":1}},
|
||||
{"timestamp":"2026-10-21T07:51:08.074651Z","instance":"d2","stickyRejectionsByReasonDelta":{"host-rate-limited":2},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":2}},
|
||||
{"timestamp":"2026-10-21T07:51:13.829283Z","instance":"d3","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:51:14.165434Z","instance":"d4","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:51:25.769048Z","instance":"d5","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:51:35.183654Z","instance":"d1","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:51:38.073759Z","instance":"d2","stickyRejectionsByReasonDelta":{"host-rate-limited":1},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":1}},
|
||||
{"timestamp":"2026-10-21T07:51:43.829359Z","instance":"d3","stickyRejectionsByReasonDelta":{"host-rate-limited":1},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":1}},
|
||||
{"timestamp":"2026-10-21T07:51:44.166698Z","instance":"d4","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:51:55.770350Z","instance":"d5","stickyRejectionsByReasonDelta":{"host-rate-limited":1},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":1}},
|
||||
{"timestamp":"2026-10-21T07:52:05.184492Z","instance":"d1","stickyRejectionsByReasonDelta":{"host-rate-limited":1},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":1}},
|
||||
{"timestamp":"2026-10-21T07:52:08.076738Z","instance":"d2","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:52:13.830929Z","instance":"d3","stickyRejectionsByReasonDelta":{"host-rate-limited":1},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":1}},
|
||||
{"timestamp":"2026-10-21T07:52:14.167578Z","instance":"d4","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:52:25.770188Z","instance":"d5","stickyRejectionsByReasonDelta":{"host-rate-limited":1},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":1}},
|
||||
{"timestamp":"2026-10-21T07:52:35.185557Z","instance":"d1","stickyRejectionsByReasonDelta":{"host-rate-limited":1},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":1}},
|
||||
{"timestamp":"2026-10-21T07:52:38.073056Z","instance":"d2","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:52:43.830612Z","instance":"d3","stickyRejectionsByReasonDelta":{"host-rate-limited":1},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":1}},
|
||||
{"timestamp":"2026-10-21T07:52:44.170744Z","instance":"d4","stickyRejectionsByReasonDelta":{"host-rate-limited":1},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":1}},
|
||||
{"timestamp":"2026-10-21T07:52:55.770072Z","instance":"d5","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:53:05.185184Z","instance":"d1","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:53:08.073799Z","instance":"d2","stickyRejectionsByReasonDelta":{"host-rate-limited":2},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":2}},
|
||||
{"timestamp":"2026-10-21T07:53:13.831126Z","instance":"d3","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:53:14.168954Z","instance":"d4","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:53:25.771642Z","instance":"d5","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:53:35.185375Z","instance":"d1","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:53:38.073871Z","instance":"d2","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:53:43.832470Z","instance":"d3","stickyRejectionsByReasonDelta":{"host-rate-limited":1},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":1}},
|
||||
{"timestamp":"2026-10-21T07:53:44.169801Z","instance":"d4","stickyRejectionsByReasonDelta":{"host-rate-limited":2},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":2}},
|
||||
{"timestamp":"2026-10-21T07:53:55.772778Z","instance":"d5","stickyRejectionsByReasonDelta":{"host-rate-limited":2},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":2}},
|
||||
{"timestamp":"2026-10-21T07:54:05.184779Z","instance":"d1","stickyRejectionsByReasonDelta":{"host-rate-limited":1},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":1}},
|
||||
{"timestamp":"2026-10-21T07:54:08.075259Z","instance":"d2","stickyRejectionsByReasonDelta":{"host-rate-limited":2},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":2}},
|
||||
{"timestamp":"2026-10-21T07:54:13.831013Z","instance":"d3","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:54:14.170568Z","instance":"d4","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:54:25.772879Z","instance":"d5","stickyRejectionsByReasonDelta":{"host-rate-limited":1},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":1}},
|
||||
{"timestamp":"2026-10-21T07:54:35.185747Z","instance":"d1","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:54:38.074242Z","instance":"d2","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:54:43.832593Z","instance":"d3","stickyRejectionsByReasonDelta":{"host-rate-limited":1},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":1}},
|
||||
{"timestamp":"2026-10-21T07:54:44.170223Z","instance":"d4","stickyRejectionsByReasonDelta":{"host-rate-limited":2},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":2}},
|
||||
{"timestamp":"2026-10-21T07:54:55.772796Z","instance":"d5","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:55:05.186732Z","instance":"d1","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:55:08.074506Z","instance":"d2","stickyRejectionsByReasonDelta":{"host-rate-limited":1},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":1}},
|
||||
{"timestamp":"2026-10-21T07:55:13.832282Z","instance":"d3","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:55:14.170168Z","instance":"d4","stickyRejectionsByReasonDelta":{"host-rate-limited":1},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":1}},
|
||||
{"timestamp":"2026-10-21T07:55:25.773007Z","instance":"d5","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:55:35.186844Z","instance":"d1","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:55:38.073584Z","instance":"d2","stickyRejectionsByReasonDelta":{"host-rate-limited":1},"placementRejectionsByReasonDelta":{"host-rate-limited":1},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":1}},
|
||||
{"timestamp":"2026-10-21T07:55:43.832666Z","instance":"d3","stickyRejectionsByReasonDelta":{"host-rate-limited":2},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":2}},
|
||||
{"timestamp":"2026-10-21T07:55:44.171703Z","instance":"d4","stickyRejectionsByReasonDelta":{"host-rate-limited":2},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":2}},
|
||||
{"timestamp":"2026-10-21T07:55:55.774228Z","instance":"d5","stickyRejectionsByReasonDelta":{"wait-timeout":21},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":21}},
|
||||
{"timestamp":"2026-10-21T07:56:05.186316Z","instance":"d1","stickyRejectionsByReasonDelta":{"host-rate-limited":4},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":4}},
|
||||
{"timestamp":"2026-10-21T07:56:08.074683Z","instance":"d2","stickyRejectionsByReasonDelta":{"host-rate-limited":1},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":1}},
|
||||
{"timestamp":"2026-10-21T07:56:13.836910Z","instance":"d3","stickyRejectionsByReasonDelta":{"host-rate-limited":2},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":2}},
|
||||
{"timestamp":"2026-10-21T07:56:14.171278Z","instance":"d4","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:56:25.774220Z","instance":"d5","stickyRejectionsByReasonDelta":{"wait-timeout":132},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":132}},
|
||||
{"timestamp":"2026-10-21T07:56:35.187168Z","instance":"d1","stickyRejectionsByReasonDelta":{"host-rate-limited":1},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":1}},
|
||||
{"timestamp":"2026-10-21T07:56:38.075749Z","instance":"d2","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:56:43.834101Z","instance":"d3","stickyRejectionsByReasonDelta":{"host-rate-limited":4},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":4}},
|
||||
{"timestamp":"2026-10-21T07:56:44.171918Z","instance":"d4","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:56:55.775731Z","instance":"d5","stickyRejectionsByReasonDelta":{"host-rate-limited":1,"wait-timeout":56},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":57}},
|
||||
{"timestamp":"2026-10-21T07:57:05.187236Z","instance":"d1","stickyRejectionsByReasonDelta":{"host-rate-limited":1},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":1}},
|
||||
{"timestamp":"2026-10-21T07:57:08.074119Z","instance":"d2","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:57:13.833738Z","instance":"d3","stickyRejectionsByReasonDelta":{"host-rate-limited":1},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":1}},
|
||||
{"timestamp":"2026-10-21T07:57:14.171816Z","instance":"d4","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:57:25.775798Z","instance":"d5","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:57:35.187493Z","instance":"d1","stickyRejectionsByReasonDelta":{"host-rate-limited":1},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":1}},
|
||||
{"timestamp":"2026-10-21T07:57:38.075303Z","instance":"d2","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:57:43.833226Z","instance":"d3","stickyRejectionsByReasonDelta":{"host-rate-limited":2},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":2}},
|
||||
{"timestamp":"2026-10-21T07:57:44.170780Z","instance":"d4","drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0},
|
||||
{"timestamp":"2026-10-21T07:57:55.775393Z","instance":"d5","stickyRejectionsByReasonDelta":{"wait-timeout":5},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":5}},
|
||||
{"timestamp":"2026-10-21T07:58:05.187113Z","instance":"d1","stickyRejectionsByReasonDelta":{"wait-timeout":9},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":9}},
|
||||
{"timestamp":"2026-10-21T07:58:08.073793Z","instance":"d2","stickyRejectionsByReasonDelta":{"host-rate-limited":1,"wait-timeout":3},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":4}},
|
||||
{"timestamp":"2026-10-21T07:58:13.849059Z","instance":"d3","stickyRejectionsByReasonDelta":{"wait-timeout":11},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":11}},
|
||||
{"timestamp":"2026-10-21T07:58:14.171116Z","instance":"d4","stickyRejectionsByReasonDelta":{"wait-timeout":13},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":13}},
|
||||
{"timestamp":"2026-10-21T07:58:25.776011Z","instance":"d5","stickyRejectionsByReasonDelta":{"host-rate-limited":1},"drainReturnDeferralsDelta":0,"drainReturnAssignmentsDelta":0,"assign503sByCauseDelta":{"sticky-lane":1}}
|
||||
]
|
||||
}
|
||||
@@ -1,339 +0,0 @@
|
||||
{
|
||||
"comment": "Terraform root ownership for every resource family declared under infra/terraform (relay), infra/terraform-foundation, and infra/terraform-apps. env_conditional families live in relay for production and in apps for staging (complementary counts). never_in_relay_state lists foundation/apps families that were introduced with their new root and so must NOT appear in infra/terraform/relay-root-carve-removed.tf. Generated by dev/scripts/terraform-root-partition.mjs --write; the test asserts this file matches the declarations.",
|
||||
"foundation": [
|
||||
"google_artifact_registry_repository.api",
|
||||
"google_iam_workload_identity_pool.github",
|
||||
"google_project_service.required",
|
||||
"google_project_service.sqladmin",
|
||||
"google_service_account.runtime",
|
||||
"google_sql_database_instance.auth",
|
||||
"google_storage_bucket_iam_member.cloud_sql_rollout_lease",
|
||||
"google_storage_bucket_iam_member.cloud_sql_rollout_lease_bucket_reader"
|
||||
],
|
||||
"apps": [
|
||||
"cloudflare_record.artifact_onorca",
|
||||
"cloudflare_record.artifact_usercontent",
|
||||
"cloudflare_record.auth",
|
||||
"google_artifact_registry_repository_iam_member.github_production_app_deploy_writer",
|
||||
"google_cloud_run_domain_mapping.artifacts",
|
||||
"google_cloud_run_domain_mapping.auth",
|
||||
"google_cloud_run_v2_job.skill_storage_monitor",
|
||||
"google_cloud_run_v2_job_iam_member.github_production_app_skill_monitor_developer",
|
||||
"google_cloud_run_v2_job_iam_member.skill_storage_scheduler_invoker",
|
||||
"google_cloud_run_v2_service.api",
|
||||
"google_cloud_run_v2_service.auth",
|
||||
"google_cloud_run_v2_service_iam_member.github_production_app_api_developer",
|
||||
"google_cloud_run_v2_service_iam_member.github_production_app_auth_developer",
|
||||
"google_cloud_scheduler_job.skill_storage_monitor",
|
||||
"google_iam_workload_identity_pool_provider.github_production_app_deploy",
|
||||
"google_logging_metric.skill_api",
|
||||
"google_logging_metric.skill_archive_rejection",
|
||||
"google_logging_metric.skill_database",
|
||||
"google_logging_metric.skill_digest_mismatch",
|
||||
"google_logging_metric.skill_finalize_saturation",
|
||||
"google_logging_metric.skill_route_latency",
|
||||
"google_logging_metric.skill_signing_failure",
|
||||
"google_logging_metric.skill_storage_inventory",
|
||||
"google_logging_metric.skill_storage_overdue_quarantine",
|
||||
"google_logging_project_exclusion.skill_share_bearer_request_urls",
|
||||
"google_monitoring_alert_policy.skill_api_5xx",
|
||||
"google_monitoring_alert_policy.skill_cloud_run_resource_pressure",
|
||||
"google_monitoring_alert_policy.skill_database_migration_failure",
|
||||
"google_monitoring_alert_policy.skill_digest_mismatch",
|
||||
"google_monitoring_alert_policy.skill_finalize_failures",
|
||||
"google_monitoring_alert_policy.skill_route_latency",
|
||||
"google_monitoring_alert_policy.skill_signing_failure",
|
||||
"google_monitoring_alert_policy.skill_storage_lifecycle_failure",
|
||||
"google_monitoring_dashboard.skill_sharing",
|
||||
"google_project_iam_custom_role.skill_storage_inventory",
|
||||
"google_project_iam_member.auth_runtime_cloudsql_client",
|
||||
"google_project_iam_member.github_artifact_writer",
|
||||
"google_project_iam_member.github_cloud_run_developer",
|
||||
"google_project_iam_member.runtime_skills_cloudsql_client",
|
||||
"google_secret_manager_secret.auth",
|
||||
"google_secret_manager_secret.auth_database_url",
|
||||
"google_secret_manager_secret.skills_database_url",
|
||||
"google_secret_manager_secret_iam_member.auth_database_url_accessor",
|
||||
"google_secret_manager_secret_iam_member.auth_runtime_accessor",
|
||||
"google_secret_manager_secret_iam_member.runtime_skills_database_url_accessor",
|
||||
"google_secret_manager_secret_version.auth_database_url",
|
||||
"google_secret_manager_secret_version.skills_database_url",
|
||||
"google_service_account.auth_runtime",
|
||||
"google_service_account.github_production_app_deploy",
|
||||
"google_service_account.skill_storage_monitor",
|
||||
"google_service_account.skill_storage_scheduler",
|
||||
"google_service_account_iam_member.github_auth_runtime_service_account_user",
|
||||
"google_service_account_iam_member.github_production_app_auth_runtime_user",
|
||||
"google_service_account_iam_member.github_production_app_deploy_workload_identity_user",
|
||||
"google_service_account_iam_member.github_production_app_runtime_user",
|
||||
"google_service_account_iam_member.github_production_app_skill_monitor_user",
|
||||
"google_service_account_iam_member.github_runtime_service_account_user",
|
||||
"google_service_account_iam_member.github_skill_storage_monitor_user",
|
||||
"google_service_account_iam_member.runtime_skill_package_signer",
|
||||
"google_sql_database.auth",
|
||||
"google_sql_database.skills",
|
||||
"google_sql_user.auth",
|
||||
"google_sql_user.skills",
|
||||
"google_storage_bucket.artifacts",
|
||||
"google_storage_bucket.skill_packages",
|
||||
"google_storage_bucket_iam_member.runtime_artifact_object_admin",
|
||||
"google_storage_bucket_iam_member.runtime_skill_package_object_user",
|
||||
"google_storage_bucket_iam_member.skill_storage_monitor_inventory",
|
||||
"random_password.auth_database",
|
||||
"random_password.skills_database",
|
||||
"time_sleep.skill_metric_descriptor_propagation"
|
||||
],
|
||||
"relay": [
|
||||
"google_artifact_registry_repository_iam_member.github_production_relay_staging_mirror_writer",
|
||||
"google_artifact_registry_repository_iam_member.github_production_relay_writer",
|
||||
"google_artifact_registry_repository_iam_member.github_push_artifact_writer",
|
||||
"google_artifact_registry_repository_iam_member.github_relay_asia_topology_artifact_reader",
|
||||
"google_artifact_registry_repository_iam_member.github_staging_relay_deploy_artifact_reader",
|
||||
"google_certificate_manager_certificate.relay_gce",
|
||||
"google_certificate_manager_certificate_map.relay_gce",
|
||||
"google_certificate_manager_certificate_map_entry.relay_gce",
|
||||
"google_certificate_manager_dns_authorization.relay_gce",
|
||||
"google_cloud_run_domain_mapping.push",
|
||||
"google_cloud_run_domain_mapping.relay",
|
||||
"google_cloud_run_domain_mapping.relay_cell",
|
||||
"google_cloud_run_v2_service.push",
|
||||
"google_cloud_run_v2_service.relay",
|
||||
"google_cloud_run_v2_service.relay_cell",
|
||||
"google_cloud_run_v2_service.relay_fence_broker",
|
||||
"google_cloud_run_v2_service_iam_member.github_production_push_developer",
|
||||
"google_cloud_run_v2_service_iam_member.github_production_relay_director_developer",
|
||||
"google_cloud_run_v2_service_iam_member.github_production_relay_fence_broker_developer",
|
||||
"google_cloud_run_v2_service_iam_member.github_staging_relay_capacity_developer",
|
||||
"google_cloud_run_v2_service_iam_member.github_staging_relay_deploy_auth_developer",
|
||||
"google_cloud_run_v2_service_iam_member.github_staging_relay_deploy_director_developer",
|
||||
"google_cloud_run_v2_service_iam_member.relay_fence_broker_deploy_invoker",
|
||||
"google_cloud_run_v2_service_iam_member.relay_fence_broker_invoker",
|
||||
"google_compute_backend_service.relay_gce_cell",
|
||||
"google_compute_firewall.relay_gce_iap_ssh",
|
||||
"google_compute_firewall.relay_gce_load_balancer",
|
||||
"google_compute_global_address.relay_gce",
|
||||
"google_compute_global_forwarding_rule.relay_gce",
|
||||
"google_compute_health_check.relay_gce_liveness",
|
||||
"google_compute_health_check.relay_gce_readiness",
|
||||
"google_compute_instance_group_manager.relay_gce_cell",
|
||||
"google_compute_instance_template.relay_gce_cell",
|
||||
"google_compute_network.relay_gce",
|
||||
"google_compute_router.relay_gce",
|
||||
"google_compute_router.relay_gce_additional",
|
||||
"google_compute_router_nat.relay_gce",
|
||||
"google_compute_router_nat.relay_gce_additional",
|
||||
"google_compute_subnetwork.relay_gce",
|
||||
"google_compute_subnetwork.relay_gce_additional",
|
||||
"google_compute_target_https_proxy.relay_gce",
|
||||
"google_compute_url_map.relay_gce",
|
||||
"google_iam_workload_identity_pool_provider.github_fence",
|
||||
"google_iam_workload_identity_pool_provider.github_monitor",
|
||||
"google_iam_workload_identity_pool_provider.github_production_relay_capacity",
|
||||
"google_iam_workload_identity_pool_provider.github_push",
|
||||
"google_iam_workload_identity_pool_provider.github_relay_asia_proof",
|
||||
"google_iam_workload_identity_pool_provider.github_relay_asia_topology",
|
||||
"google_iam_workload_identity_pool_provider.github_staging_relay_capacity",
|
||||
"google_iam_workload_identity_pool_provider.github_staging_relay_deploy",
|
||||
"google_logging_metric.relay_assignment_lease_shadow",
|
||||
"google_logging_metric.relay_cell_fix_level",
|
||||
"google_logging_metric.relay_cell_pool_herd",
|
||||
"google_logging_metric.relay_cell_reregistration_gave_up",
|
||||
"google_logging_metric.relay_cell_reregistration_stalled",
|
||||
"google_logging_metric.relay_cell_reserve_dead_man_tripped",
|
||||
"google_logging_metric.relay_control_flags_unreadable",
|
||||
"google_logging_metric.relay_dead_cell_verdict_withheld",
|
||||
"google_logging_metric.relay_incident",
|
||||
"google_logging_metric.relay_event_loop_stall",
|
||||
"google_logging_metric.relay_long_cell_inventory_hold",
|
||||
"google_logging_metric.relay_reserve_placement_off_with_reserve_cells",
|
||||
"google_logging_metric.relay_reserve_watch_alarm",
|
||||
"google_logging_metric.relay_snapshot",
|
||||
"google_monitoring_alert_policy.relay_assignment_5xx",
|
||||
"google_monitoring_alert_policy.relay_assignment_edge_429",
|
||||
"google_monitoring_alert_policy.relay_assignment_lease_bad_signature",
|
||||
"google_monitoring_alert_policy.relay_cell_control_close_burst",
|
||||
"google_monitoring_alert_policy.relay_cell_control_rtt",
|
||||
"google_monitoring_alert_policy.relay_cell_outdated_fix_level",
|
||||
"google_monitoring_alert_policy.relay_cell_pool_herd",
|
||||
"google_monitoring_alert_policy.relay_cell_reregistration_gave_up",
|
||||
"google_monitoring_alert_policy.relay_cell_reregistration_stalled",
|
||||
"google_monitoring_alert_policy.relay_cell_reserve_dead_man_tripped",
|
||||
"google_monitoring_alert_policy.relay_cell_process_exit",
|
||||
"google_monitoring_alert_policy.relay_cloud_nat_port_drops",
|
||||
"google_monitoring_alert_policy.relay_cloud_sql_backends",
|
||||
"google_monitoring_alert_policy.relay_cloud_sql_checkpoint_loop",
|
||||
"google_monitoring_alert_policy.relay_cloud_sql_disk",
|
||||
"google_monitoring_alert_policy.relay_control_flags_unreadable",
|
||||
"google_monitoring_alert_policy.relay_custom",
|
||||
"google_monitoring_alert_policy.relay_database_rejection_fenced",
|
||||
"google_monitoring_alert_policy.relay_dead_cell_verdict_withheld",
|
||||
"google_monitoring_alert_policy.relay_director_cell_inventory_hold",
|
||||
"google_monitoring_alert_policy.relay_far_cell_accept_latency",
|
||||
"google_monitoring_alert_policy.relay_gce_connection_headroom",
|
||||
"google_monitoring_alert_policy.relay_lock_timeout_burst",
|
||||
"google_monitoring_alert_policy.relay_event_loop_stall",
|
||||
"google_monitoring_alert_policy.relay_long_cell_inventory_hold",
|
||||
"google_monitoring_alert_policy.relay_postgres_retry_exhausted",
|
||||
"google_monitoring_alert_policy.relay_process_fatal",
|
||||
"google_monitoring_alert_policy.relay_region_hint_skew",
|
||||
"google_monitoring_alert_policy.relay_reserve_placement_off_with_reserve_cells",
|
||||
"google_monitoring_alert_policy.relay_reserve_watch_alarm",
|
||||
"google_monitoring_dashboard.relay_incident",
|
||||
"google_project_iam_custom_role.github_production_relay_capacity_mutation",
|
||||
"google_project_iam_custom_role.github_relay_asia_topology_mutation",
|
||||
"google_project_iam_custom_role.github_relay_asia_topology_read",
|
||||
"google_project_iam_custom_role.github_relay_asia_topology_state_list",
|
||||
"google_project_iam_custom_role.github_staging_relay_capacity_mutation",
|
||||
"google_project_iam_custom_role.github_staging_relay_power",
|
||||
"google_project_iam_custom_role.relay_control_writer",
|
||||
"google_project_iam_custom_role.relay_fence_broker_mutation",
|
||||
"google_project_iam_member.github_fence_cloudsql_viewer",
|
||||
"google_project_iam_member.github_fence_compute_viewer",
|
||||
"google_project_iam_member.github_fence_logging_viewer",
|
||||
"google_project_iam_member.github_fence_monitoring_viewer",
|
||||
"google_project_iam_member.github_monitor_compute_viewer",
|
||||
"google_project_iam_member.github_production_relay_capacity_artifact_reader",
|
||||
"google_project_iam_member.github_production_relay_capacity_mutation",
|
||||
"google_project_iam_member.github_production_relay_capacity_viewer",
|
||||
"google_project_iam_member.github_relay_asia_proof_logging_viewer",
|
||||
"google_project_iam_member.github_relay_asia_proof_monitoring_viewer",
|
||||
"google_project_iam_member.github_relay_asia_topology_mutation",
|
||||
"google_project_iam_member.github_relay_asia_topology_read",
|
||||
"google_project_iam_member.github_relay_monitor_cloudsql_viewer",
|
||||
"google_project_iam_member.github_relay_monitor_logging_viewer",
|
||||
"google_project_iam_member.github_relay_monitor_monitoring_viewer",
|
||||
"google_project_iam_member.github_staging_relay_capacity_artifact_reader",
|
||||
"google_project_iam_member.github_staging_relay_capacity_mutation",
|
||||
"google_project_iam_member.github_staging_relay_capacity_viewer",
|
||||
"google_project_iam_member.github_staging_relay_deploy_compute_viewer",
|
||||
"google_project_iam_member.github_staging_relay_deploy_logging_viewer",
|
||||
"google_project_iam_member.github_staging_relay_power",
|
||||
"google_project_iam_member.push_runtime_cloudsql_client",
|
||||
"google_project_iam_member.push_runtime_fcm_admin",
|
||||
"google_project_iam_member.push_runtime_service_usage_consumer",
|
||||
"google_project_iam_member.relay_director_runtime_cloudsql_client",
|
||||
"google_project_iam_member.relay_fence_broker_artifact_reader",
|
||||
"google_project_iam_member.relay_fence_broker_compute_viewer",
|
||||
"google_project_iam_member.relay_fence_broker_logging_viewer",
|
||||
"google_project_iam_member.relay_fence_broker_mutation",
|
||||
"google_project_iam_member.relay_runtime_artifact_reader",
|
||||
"google_project_iam_member.relay_runtime_cloudsql_client",
|
||||
"google_project_iam_member.relay_runtime_log_writer",
|
||||
"google_secret_manager_secret.push_dedicated_database_url",
|
||||
"google_secret_manager_secret.push_provider",
|
||||
"google_secret_manager_secret.relay_assignment_signing_key",
|
||||
"google_secret_manager_secret.relay_database_url",
|
||||
"google_secret_manager_secret.relay_director_database_url",
|
||||
"google_secret_manager_secret.relay_regional_placement_enabled",
|
||||
"google_secret_manager_secret_iam_member.push_dedicated_database_url_accessor",
|
||||
"google_secret_manager_secret_iam_member.push_provider_runtime_accessor",
|
||||
"google_secret_manager_secret_iam_member.relay_assignment_signing_key_accessor",
|
||||
"google_secret_manager_secret_iam_member.relay_assignment_signing_key_director_accessor",
|
||||
"google_secret_manager_secret_iam_member.relay_database_url_accessor",
|
||||
"google_secret_manager_secret_iam_member.relay_database_url_director_accessor",
|
||||
"google_secret_manager_secret_iam_member.relay_director_database_url_accessor",
|
||||
"google_secret_manager_secret_iam_member.relay_regional_placement_deploy_accessor",
|
||||
"google_secret_manager_secret_iam_member.relay_regional_placement_deploy_adder",
|
||||
"google_secret_manager_secret_iam_member.relay_regional_placement_deploy_viewer",
|
||||
"google_secret_manager_secret_iam_member.relay_regional_placement_director_accessor",
|
||||
"google_secret_manager_secret_iam_member.relay_regional_placement_runtime_accessor",
|
||||
"google_secret_manager_secret_version.push_dedicated_database_url",
|
||||
"google_secret_manager_secret_version.relay_assignment_signing_key",
|
||||
"google_secret_manager_secret_version.relay_database_url",
|
||||
"google_secret_manager_secret_version.relay_director_database_url",
|
||||
"google_secret_manager_secret_version.relay_regional_placement_enabled",
|
||||
"google_service_account.github_fence",
|
||||
"google_service_account.github_monitor",
|
||||
"google_service_account.github_production_relay_capacity",
|
||||
"google_service_account.github_push_deploy",
|
||||
"google_service_account.github_relay_asia_proof",
|
||||
"google_service_account.github_relay_asia_topology",
|
||||
"google_service_account.github_staging_relay_capacity",
|
||||
"google_service_account.github_staging_relay_deploy",
|
||||
"google_service_account.push_runtime",
|
||||
"google_service_account.relay_director_runtime",
|
||||
"google_service_account.relay_fence_broker",
|
||||
"google_service_account.relay_runtime",
|
||||
"google_service_account_iam_member.github_accepted_repository_workload_identity_user",
|
||||
"google_service_account_iam_member.github_fence_workload_identity_user",
|
||||
"google_service_account_iam_member.github_monitor_workload_identity_user",
|
||||
"google_service_account_iam_member.github_production_push_runtime_token_creator",
|
||||
"google_service_account_iam_member.github_production_push_runtime_user",
|
||||
"google_service_account_iam_member.github_production_relay_capacity_runtime_user",
|
||||
"google_service_account_iam_member.github_production_relay_capacity_workload_identity_user",
|
||||
"google_service_account_iam_member.github_push_workload_identity_user",
|
||||
"google_service_account_iam_member.github_relay_asia_proof_workload_identity_user",
|
||||
"google_service_account_iam_member.github_relay_asia_topology_runtime_user",
|
||||
"google_service_account_iam_member.github_relay_asia_topology_workload_identity_user",
|
||||
"google_service_account_iam_member.github_relay_director_runtime_service_account_user",
|
||||
"google_service_account_iam_member.github_relay_fence_broker_service_account_user",
|
||||
"google_service_account_iam_member.github_relay_runtime_service_account_user",
|
||||
"google_service_account_iam_member.github_staging_relay_capacity_runtime_user",
|
||||
"google_service_account_iam_member.github_staging_relay_capacity_workload_identity_user",
|
||||
"google_service_account_iam_member.github_staging_relay_deploy_auth_runtime_user",
|
||||
"google_service_account_iam_member.github_staging_relay_deploy_workload_identity_user",
|
||||
"google_service_account_iam_member.relay_fence_broker_requester_token_creator",
|
||||
"google_sql_database.push_dedicated",
|
||||
"google_sql_database.relay",
|
||||
"google_sql_database_instance.push_dedicated",
|
||||
"google_sql_user.push_dedicated",
|
||||
"google_sql_user.relay",
|
||||
"google_storage_bucket.relay_control",
|
||||
"google_storage_bucket_iam_member.github_production_relay_capacity_state",
|
||||
"google_storage_bucket_iam_member.github_push_rollout_lease",
|
||||
"google_storage_bucket_iam_member.github_relay_asia_topology_state",
|
||||
"google_storage_bucket_iam_member.github_relay_asia_topology_state_list",
|
||||
"google_storage_bucket_iam_member.github_staging_relay_capacity_state",
|
||||
"google_storage_bucket_iam_member.github_staging_relay_deploy_state",
|
||||
"google_storage_bucket_iam_member.github_staging_relay_deploy_state_list",
|
||||
"google_storage_bucket_iam_member.relay_control_cell_reader",
|
||||
"google_storage_bucket_iam_member.relay_control_workflow_writer",
|
||||
"google_storage_bucket_iam_member.relay_fence_broker_bucket_reader",
|
||||
"google_storage_bucket_iam_member.relay_fence_broker_state_objects",
|
||||
"random_password.push_dedicated_database",
|
||||
"random_password.relay_assignment_signing_key",
|
||||
"random_password.relay_database"
|
||||
],
|
||||
"env_conditional": {
|
||||
"google_iam_workload_identity_pool_provider.github": {
|
||||
"production": "relay",
|
||||
"staging": "apps"
|
||||
},
|
||||
"google_project_iam_member.github_cloudsql_viewer": {
|
||||
"production": "relay",
|
||||
"staging": "apps"
|
||||
},
|
||||
"google_project_iam_member.github_compute_viewer": {
|
||||
"production": "relay",
|
||||
"staging": "apps"
|
||||
},
|
||||
"google_project_iam_member.github_log_writer": {
|
||||
"production": "relay",
|
||||
"staging": "apps"
|
||||
},
|
||||
"google_project_iam_member.github_logging_viewer": {
|
||||
"production": "relay",
|
||||
"staging": "apps"
|
||||
},
|
||||
"google_project_iam_member.github_monitoring_viewer": {
|
||||
"production": "relay",
|
||||
"staging": "apps"
|
||||
},
|
||||
"google_service_account.github_deploy": {
|
||||
"production": "relay",
|
||||
"staging": "apps"
|
||||
},
|
||||
"google_service_account_iam_member.github_workload_identity_user": {
|
||||
"production": "relay",
|
||||
"staging": "apps"
|
||||
},
|
||||
"google_storage_bucket_iam_member.github_terraform_state_reader": {
|
||||
"production": "relay",
|
||||
"staging": "apps"
|
||||
}
|
||||
},
|
||||
"state_orphans": {
|
||||
"staging": [],
|
||||
"production": []
|
||||
}
|
||||
}
|
||||
@@ -1,83 +0,0 @@
|
||||
#!/usr/bin/env node
|
||||
import { execFileSync } from 'node:child_process'
|
||||
import { mkdirSync, readFileSync, writeFileSync } from 'node:fs'
|
||||
import { join } from 'node:path'
|
||||
|
||||
// Why: the relay root can never plan zero-diff (cell templates roll one at a time by design),
|
||||
// so the split is gated on plan EQUIVALENCE: the normalized change set for a root's addresses
|
||||
// must be identical before and after a state move. This captures that set deterministically.
|
||||
// Read-only: -lock=false, -refresh=false, no apply. Forgets from `removed` blocks are excluded
|
||||
// because the baseline has none.
|
||||
|
||||
const usage =
|
||||
'usage: capture-terraform-plan-baseline.mjs --root <dir> --env <staging|production> --out <dir> [--tag <name>]'
|
||||
|
||||
export function normalizePlan(planJson) {
|
||||
const changes = (planJson.resource_changes ?? [])
|
||||
.filter((entry) => !entry.change.actions.includes('forget'))
|
||||
.map((entry) => ({
|
||||
address: entry.address,
|
||||
actions: entry.change.actions,
|
||||
before: entry.change.before ?? null,
|
||||
after: entry.change.after ?? null,
|
||||
after_unknown: entry.change.after_unknown ?? null
|
||||
}))
|
||||
.sort((left, right) => (left.address < right.address ? -1 : left.address > right.address ? 1 : 0))
|
||||
return changes
|
||||
}
|
||||
|
||||
export function summarize(changes) {
|
||||
const counts = { create: 0, update: 0, delete: 0, replace: 0, 'no-op': 0, read: 0 }
|
||||
for (const change of changes) {
|
||||
const key = change.actions.join('-')
|
||||
if (key === 'create') counts.create += 1
|
||||
else if (key === 'update') counts.update += 1
|
||||
else if (key === 'delete') counts.delete += 1
|
||||
else if (key === 'delete-create' || key === 'create-delete') counts.replace += 1
|
||||
else if (key === 'read') counts.read += 1
|
||||
else counts['no-op'] += 1
|
||||
}
|
||||
return counts
|
||||
}
|
||||
|
||||
function argument(flag) {
|
||||
const index = process.argv.indexOf(flag)
|
||||
return index >= 0 ? process.argv[index + 1] : undefined
|
||||
}
|
||||
|
||||
if (process.argv[1] && import.meta.url.endsWith(process.argv[1].split('/').pop())) {
|
||||
const root = argument('--root')
|
||||
const environment = argument('--env')
|
||||
const out = argument('--out')
|
||||
const tag = argument('--tag') ?? `${environment}-${root.replaceAll('/', '_')}`
|
||||
if (!root || !['staging', 'production'].includes(environment) || !out) {
|
||||
process.stderr.write(`${usage}\n`)
|
||||
process.exit(2)
|
||||
}
|
||||
// The Cloudflare override only applies to the root that still declares the records; the relay
|
||||
// root dropped them in the carve and errors on a -var for an undeclared variable.
|
||||
const declaresArtifactDns = readFileSync(join(root, 'variables.tf'), 'utf8').includes(
|
||||
'variable "manage_artifact_dns"'
|
||||
)
|
||||
mkdirSync(out, { recursive: true })
|
||||
const planFile = join(out, `${tag}.tfplan`)
|
||||
execFileSync(
|
||||
'terraform',
|
||||
[
|
||||
`-chdir=${root}`, 'plan', '-input=false', '-lock=false', '-refresh=false', '-no-color',
|
||||
`-var-file=environments/${environment}.tfvars`,
|
||||
...(declaresArtifactDns ? ['-var', 'manage_artifact_dns=false'] : []),
|
||||
`-out=${planFile}`
|
||||
],
|
||||
{ stdio: ['ignore', 'inherit', 'inherit'] }
|
||||
)
|
||||
const json = JSON.parse(
|
||||
execFileSync('terraform', [`-chdir=${root}`, 'show', '-json', planFile], {
|
||||
encoding: 'utf8',
|
||||
maxBuffer: 256 * 1024 * 1024
|
||||
})
|
||||
)
|
||||
const normalized = normalizePlan(json)
|
||||
writeFileSync(join(out, `${tag}.norm.json`), `${JSON.stringify(normalized, null, 1)}\n`)
|
||||
process.stdout.write(`${tag}: ${JSON.stringify(summarize(normalized))}\n`)
|
||||
}
|
||||
@@ -1,15 +0,0 @@
|
||||
import assert from 'node:assert/strict'
|
||||
import { test } from 'node:test'
|
||||
import { normalizePlan, summarize } from './capture-terraform-plan-baseline.mjs'
|
||||
|
||||
test('normalizes, sorts, and drops forgets so a removed block cannot skew equivalence', () => {
|
||||
const changes = normalizePlan({
|
||||
resource_changes: [
|
||||
{ address: 'b.two', change: { actions: ['update'], before: { x: 1 }, after: { x: 2 } } },
|
||||
{ address: 'a.one', change: { actions: ['forget'], before: {}, after: null } },
|
||||
{ address: 'c.three', change: { actions: ['delete', 'create'], before: {}, after: {}, after_unknown: { id: true } } }
|
||||
]
|
||||
})
|
||||
assert.deepEqual(changes.map((change) => change.address), ['b.two', 'c.three'])
|
||||
assert.deepEqual(summarize(changes), { create: 0, update: 1, delete: 0, replace: 1, 'no-op': 0, read: 0 })
|
||||
})
|
||||
@@ -1,167 +0,0 @@
|
||||
import { pathToFileURL } from 'node:url'
|
||||
import { fetchAdminOnceMore } from './relay-admin-transient-retry.mjs'
|
||||
import { isReserveCellRefusal, reserveCellRollRefusal } from './relay-reserve-cell-refusal.mjs'
|
||||
|
||||
// Drained hosts that find no free slot keep redialling and pin the drained cell (c28,
|
||||
// 2026-10-01). Hosts are counted as controls, but each moved host also brings its splices;
|
||||
// the 20% margin is for those.
|
||||
export const MAX_HEADROOM_FRACTION = 0.8
|
||||
|
||||
function count(value, name) {
|
||||
if (!Number.isSafeInteger(value) || value < 0) throw new Error(`${name} is invalid`)
|
||||
return value
|
||||
}
|
||||
|
||||
export function parseHeadroomArguments(argv) {
|
||||
const values = {}
|
||||
for (let index = 0; index < argv.length; index += 2) {
|
||||
const key = argv[index]
|
||||
const value = argv[index + 1]
|
||||
if (!key?.startsWith('--') || value === undefined) throw new Error('invalid arguments')
|
||||
values[key.slice(2)] = value
|
||||
}
|
||||
for (const key of ['director-origin', 'cell-origin', 'cell-id', 'general-cells']) {
|
||||
if (!values[key]) throw new Error(`missing --${key}`)
|
||||
}
|
||||
for (const key of ['director-origin', 'cell-origin']) {
|
||||
const url = new URL(values[key])
|
||||
if (url.protocol !== 'https:' || url.origin !== values[key]) {
|
||||
throw new Error('origins must be canonical HTTPS origins')
|
||||
}
|
||||
}
|
||||
const generalCells = values['general-cells'] === 'none'
|
||||
? []
|
||||
: values['general-cells'].split(',')
|
||||
if (
|
||||
new Set(generalCells).size !== generalCells.length ||
|
||||
generalCells.some((cellId) => !/^[a-z0-9-]+$/.test(cellId))
|
||||
) {
|
||||
throw new Error('--general-cells is invalid')
|
||||
}
|
||||
return {
|
||||
directorOrigin: values['director-origin'],
|
||||
cellOrigin: values['cell-origin'],
|
||||
cellId: values['cell-id'],
|
||||
generalCells
|
||||
}
|
||||
}
|
||||
|
||||
async function adminJson(fetchImpl, url, token, body, wait) {
|
||||
const response = await fetchAdminOnceMore(
|
||||
fetchImpl,
|
||||
url,
|
||||
{
|
||||
method: 'POST',
|
||||
headers: { authorization: `Bearer ${token}`, 'content-type': 'application/json' },
|
||||
body: JSON.stringify(body)
|
||||
},
|
||||
{ wait }
|
||||
)
|
||||
const json = await response.json().catch(() => ({}))
|
||||
// null: the director refused the read because the cell is reserve.
|
||||
if (isReserveCellRefusal(response.status, json)) return null
|
||||
if (!response.ok) throw new Error(`${new URL(url).pathname} returned ${response.status}`)
|
||||
return json
|
||||
}
|
||||
|
||||
// A cell whose capacity view is stale, absent, or not general offers no slot we can count.
|
||||
// Placement admits while enforced units plus outstanding reservations stay under the pause.
|
||||
function freeSlots(status) {
|
||||
const capacity = status.connectionCapacity
|
||||
if (
|
||||
status.enabled !== true ||
|
||||
status.admissionState !== 'general' ||
|
||||
capacity === null ||
|
||||
capacity === undefined ||
|
||||
capacity.heartbeatFresh !== true
|
||||
) {
|
||||
return 0
|
||||
}
|
||||
const pause = count(capacity.normalAdmissionPause, 'normal admission pause')
|
||||
const used = Math.max(
|
||||
count(capacity.observedConnections, 'observed connections'),
|
||||
count(capacity.enforcedConnectionUnits, 'enforced connection units')
|
||||
)
|
||||
const reserved = count(capacity.pendingControlReservations, 'pending control reservations')
|
||||
return Math.max(0, pause - used - reserved)
|
||||
}
|
||||
|
||||
export async function checkSameCapHeadroom(config, overrides = {}) {
|
||||
const fetchImpl = overrides.fetch ?? fetch
|
||||
const wait = overrides.wait ?? ((ms) => new Promise((resolve) => setTimeout(resolve, ms)))
|
||||
const token = overrides.token ?? process.env.ORCA_RELAY_ADMIN_ID_TOKEN
|
||||
if (!token || token.length > 8_192) throw new Error('admin identity token is unavailable')
|
||||
const cellStatus = async (cellId) => {
|
||||
const answer = await adminJson(
|
||||
fetchImpl,
|
||||
`${config.directorOrigin}/v1/admin/cell-status`,
|
||||
token,
|
||||
{ v: 1, cellId },
|
||||
wait
|
||||
)
|
||||
if (answer === null) return null
|
||||
if (answer.status?.cellId !== cellId) {
|
||||
throw new Error('director status does not match the cell')
|
||||
}
|
||||
return answer.status
|
||||
}
|
||||
const runtime = await adminJson(
|
||||
fetchImpl,
|
||||
`${config.cellOrigin}/v1/admin/runtime-status`,
|
||||
token,
|
||||
{ v: 1 },
|
||||
wait
|
||||
)
|
||||
if (runtime?.role !== 'cell' || runtime.cellId !== config.cellId) {
|
||||
throw new Error('runtime status does not match the cell')
|
||||
}
|
||||
const targetHosts = count(runtime.runtime?.controls, 'target host controls')
|
||||
const targetStatus = await cellStatus(config.cellId)
|
||||
if (targetStatus === null) throw reserveCellRollRefusal(config.cellId)
|
||||
const targetRegion = targetStatus.region ?? null
|
||||
const cells = []
|
||||
for (const cellId of config.generalCells) {
|
||||
if (cellId === config.cellId) continue
|
||||
const status = await cellStatus(cellId)
|
||||
// A reserve cell's free seats are not in the database: counted as none, never guessed.
|
||||
cells.push(
|
||||
status === null
|
||||
? { cellId, region: null, freeSlots: 0, reserve: true }
|
||||
: { cellId, region: status.region ?? null, freeSlots: freeSlots(status) }
|
||||
)
|
||||
}
|
||||
const total = (filter) =>
|
||||
cells.filter(filter).reduce((sum, cell) => sum + cell.freeSlots, 0)
|
||||
const sameRegion = (cell) => targetRegion !== null && cell.region === targetRegion
|
||||
const free = total(() => true)
|
||||
const allowedHosts = Math.floor(free * MAX_HEADROOM_FRACTION)
|
||||
return {
|
||||
cellId: config.cellId,
|
||||
region: targetRegion,
|
||||
targetHosts,
|
||||
freeSlots: free,
|
||||
sameRegionFreeSlots: total(sameRegion),
|
||||
otherRegionFreeSlots: total((cell) => !sameRegion(cell)),
|
||||
allowedHosts,
|
||||
sufficient: targetHosts <= allowedHosts,
|
||||
cells
|
||||
}
|
||||
}
|
||||
|
||||
export async function main(argv = process.argv.slice(2)) {
|
||||
const result = await checkSameCapHeadroom(parseHeadroomArguments(argv))
|
||||
process.stdout.write(`${JSON.stringify({ event: 'relay_same_cap_headroom', ...result })}\n`)
|
||||
if (!result.sufficient) {
|
||||
throw new Error(
|
||||
`${result.targetHosts} hosts on ${result.cellId} exceed ` +
|
||||
`${MAX_HEADROOM_FRACTION * 100}% of ${result.freeSlots} free general-cell slots`
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
if (import.meta.url === pathToFileURL(process.argv[1]).href) {
|
||||
main().catch((error) => {
|
||||
process.stderr.write(`${error instanceof Error ? error.message : String(error)}\n`)
|
||||
process.exitCode = 1
|
||||
})
|
||||
}
|
||||
@@ -1,230 +0,0 @@
|
||||
import assert from 'node:assert/strict'
|
||||
import { test } from 'node:test'
|
||||
import {
|
||||
checkSameCapHeadroom,
|
||||
main,
|
||||
parseHeadroomArguments
|
||||
} from './check-relay-same-cap-headroom.mjs'
|
||||
|
||||
const config = {
|
||||
directorOrigin: 'https://relay.example.com',
|
||||
cellOrigin: 'https://c28.relay.example.com',
|
||||
cellId: 'production-gce-c28',
|
||||
generalCells: ['production-gce-c28', 'production-gce-c29', 'production-gce-asia-c1']
|
||||
}
|
||||
|
||||
function cell({
|
||||
region = 'us-central1',
|
||||
pause = 840,
|
||||
observed = 0,
|
||||
enforced = observed,
|
||||
pending = 0,
|
||||
...rest
|
||||
} = {}) {
|
||||
return {
|
||||
enabled: true,
|
||||
admissionState: 'general',
|
||||
region,
|
||||
connectionCapacity: {
|
||||
normalAdmissionPause: pause,
|
||||
observedConnections: observed,
|
||||
enforcedConnectionUnits: enforced,
|
||||
pendingControlReservations: pending,
|
||||
heartbeatFresh: true
|
||||
},
|
||||
...rest
|
||||
}
|
||||
}
|
||||
|
||||
function harness({ controls = 500, cells = {} } = {}) {
|
||||
const statuses = {
|
||||
'production-gce-c28': cell({ observed: 800 }),
|
||||
'production-gce-c29': cell({ observed: 340 }),
|
||||
'production-gce-asia-c1': cell({ region: 'asia-east2', pause: 2_840, observed: 2_640 }),
|
||||
...cells
|
||||
}
|
||||
return async (url, options) => {
|
||||
if (new URL(url).pathname === '/v1/admin/runtime-status') {
|
||||
return Response.json({ role: 'cell', cellId: config.cellId, runtime: { controls } })
|
||||
}
|
||||
const { cellId } = JSON.parse(options.body)
|
||||
return Response.json({ v: 1, status: { cellId, ...statuses[cellId] } })
|
||||
}
|
||||
}
|
||||
|
||||
const run = async (fetch) =>
|
||||
await checkSameCapHeadroom(config, { fetch, token: 'masked-token', wait: async () => {} })
|
||||
|
||||
test('counts free general-cell slots by region and excludes the target', async () => {
|
||||
const result = await run(harness({ controls: 560 }))
|
||||
assert.deepEqual(
|
||||
{ ...result, cells: undefined },
|
||||
{
|
||||
cellId: config.cellId,
|
||||
region: 'us-central1',
|
||||
targetHosts: 560,
|
||||
freeSlots: 700,
|
||||
sameRegionFreeSlots: 500,
|
||||
otherRegionFreeSlots: 200,
|
||||
allowedHosts: 560,
|
||||
sufficient: true,
|
||||
cells: undefined
|
||||
}
|
||||
)
|
||||
assert.deepEqual(result.cells.map((entry) => entry.cellId), [
|
||||
'production-gce-c29',
|
||||
'production-gce-asia-c1'
|
||||
])
|
||||
})
|
||||
|
||||
test('free slots subtract the larger of observed and enforced use plus reservations', async () => {
|
||||
for (const [state, free] of [
|
||||
[{ observed: 340, enforced: 400 }, 640],
|
||||
[{ observed: 400, enforced: 340 }, 640],
|
||||
[{ observed: 340, pending: 60 }, 640],
|
||||
[{ observed: 800, pending: 60 }, 200]
|
||||
]) {
|
||||
const result = await run(harness({ cells: { 'production-gce-c29': cell(state) } }))
|
||||
assert.equal(result.freeSlots, free)
|
||||
}
|
||||
})
|
||||
|
||||
test('rejects malformed capacity counts', async () => {
|
||||
await assert.rejects(
|
||||
run(harness({
|
||||
cells: { 'production-gce-c29': cell({ pending: null }) }
|
||||
})),
|
||||
/pending control reservations is invalid/
|
||||
)
|
||||
})
|
||||
|
||||
test('refuses a cell whose hosts exceed 80% of the free slots', async () => {
|
||||
const result = await run(harness({ controls: 561 }))
|
||||
assert.equal(result.sufficient, false)
|
||||
})
|
||||
|
||||
test('a stale, full, or non-general cell offers no slots', async () => {
|
||||
for (const blocked of [
|
||||
cell({
|
||||
connectionCapacity: {
|
||||
normalAdmissionPause: 840,
|
||||
observedConnections: 0,
|
||||
enforcedConnectionUnits: 0,
|
||||
pendingControlReservations: 0,
|
||||
heartbeatFresh: false
|
||||
}
|
||||
}),
|
||||
cell({ connectionCapacity: null }),
|
||||
cell({ admissionState: 'migration-only' }),
|
||||
cell({ enabled: false }),
|
||||
cell({ observed: 900 })
|
||||
]) {
|
||||
const result = await run(harness({ cells: { 'production-gce-c29': blocked } }))
|
||||
assert.equal(result.freeSlots, 200)
|
||||
assert.equal(result.sufficient, false)
|
||||
}
|
||||
})
|
||||
|
||||
test('no general cells means no headroom', async () => {
|
||||
const result = await checkSameCapHeadroom(
|
||||
{ ...config, generalCells: [] },
|
||||
{ fetch: harness({ controls: 1 }), token: 'masked-token', wait: async () => {} }
|
||||
)
|
||||
assert.equal(result.freeSlots, 0)
|
||||
assert.equal(result.sufficient, false)
|
||||
})
|
||||
|
||||
const reserveRefusal = () => Response.json({ error: 'cell_in_reserve_mode' }, { status: 404 })
|
||||
|
||||
test('a reserve cell that is not being rolled offers no slots and is named', async () => {
|
||||
const base = harness({ controls: 160 })
|
||||
const result = await run(async (url, options) =>
|
||||
JSON.parse(options.body).cellId === 'production-gce-c29'
|
||||
? reserveRefusal()
|
||||
: await base(url, options))
|
||||
assert.equal(result.freeSlots, 200)
|
||||
assert.equal(result.sufficient, true)
|
||||
assert.deepEqual(result.cells.find((entry) => entry.cellId === 'production-gce-c29'), {
|
||||
cellId: 'production-gce-c29',
|
||||
region: null,
|
||||
freeSlots: 0,
|
||||
reserve: true
|
||||
})
|
||||
})
|
||||
|
||||
test('refuses to roll a reserve cell, naming the flip back to db', async () => {
|
||||
const base = harness()
|
||||
await assert.rejects(
|
||||
run(async (url, options) =>
|
||||
JSON.parse(options.body ?? '{}').cellId === config.cellId
|
||||
? reserveRefusal()
|
||||
: await base(url, options)),
|
||||
/production-gce-c28 is in reserve mode .*flip it to admitMode=db/
|
||||
)
|
||||
})
|
||||
|
||||
test('any other 404 from cell-status still fails', async () => {
|
||||
const base = harness()
|
||||
await assert.rejects(
|
||||
run(async (url, options) =>
|
||||
JSON.parse(options.body).cellId === 'production-gce-c29'
|
||||
? Response.json({ error: 'cell_not_found' }, { status: 404 })
|
||||
: await base(url, options)),
|
||||
/cell-status returned 404/
|
||||
)
|
||||
})
|
||||
|
||||
test('rejects a runtime or director answer for another cell', async () => {
|
||||
const base = harness()
|
||||
await assert.rejects(
|
||||
run(async (url, options) =>
|
||||
new URL(url).pathname === '/v1/admin/runtime-status'
|
||||
? Response.json({ role: 'cell', cellId: 'production-gce-c1', runtime: { controls: 1 } })
|
||||
: await base(url, options)),
|
||||
/runtime status does not match/
|
||||
)
|
||||
await assert.rejects(
|
||||
run(async (url, options) =>
|
||||
new URL(url).pathname === '/v1/admin/cell-status'
|
||||
? Response.json({ v: 1, status: { cellId: 'production-gce-c1' } })
|
||||
: await base(url, options)),
|
||||
/director status does not match/
|
||||
)
|
||||
})
|
||||
|
||||
test('parses the general cell list, including none', () => {
|
||||
const base = [
|
||||
'--director-origin', 'https://relay.example.com',
|
||||
'--cell-origin', 'https://c28.relay.example.com',
|
||||
'--cell-id', 'production-gce-c28'
|
||||
]
|
||||
assert.deepEqual(
|
||||
parseHeadroomArguments([...base, '--general-cells', 'a-c1,b-c2']).generalCells,
|
||||
['a-c1', 'b-c2']
|
||||
)
|
||||
assert.deepEqual(parseHeadroomArguments([...base, '--general-cells', 'none']).generalCells, [])
|
||||
assert.throws(() => parseHeadroomArguments([...base, '--general-cells', 'a,a']), /invalid/)
|
||||
assert.throws(() => parseHeadroomArguments(base), /missing --general-cells/)
|
||||
})
|
||||
|
||||
test('main prints the numbers and fails on insufficient headroom', async (t) => {
|
||||
const writes = []
|
||||
t.mock.method(process.stdout, 'write', (text) => writes.push(text))
|
||||
t.mock.method(globalThis, 'fetch', harness({ controls: 700 }))
|
||||
process.env.ORCA_RELAY_ADMIN_ID_TOKEN = 'masked-token'
|
||||
t.after(() => {
|
||||
delete process.env.ORCA_RELAY_ADMIN_ID_TOKEN
|
||||
})
|
||||
await assert.rejects(
|
||||
main([
|
||||
'--director-origin', config.directorOrigin,
|
||||
'--cell-origin', config.cellOrigin,
|
||||
'--cell-id', config.cellId,
|
||||
'--general-cells', config.generalCells.join(',')
|
||||
]),
|
||||
/700 hosts on production-gce-c28 exceed 80% of 700 free general-cell slots/
|
||||
)
|
||||
const line = JSON.parse(writes.join(''))
|
||||
assert.equal(line.event, 'relay_same_cap_headroom')
|
||||
assert.equal(line.allowedHosts, 560)
|
||||
})
|
||||
@@ -1,109 +0,0 @@
|
||||
import { readFileSync } from 'node:fs'
|
||||
import { pathToFileURL } from 'node:url'
|
||||
import { isDeepStrictEqual } from 'node:util'
|
||||
|
||||
function parseArguments(argv) {
|
||||
if (argv.length !== 2 || argv[0] !== '--capacity-service-account' || !argv[1]) {
|
||||
throw new Error('missing --capacity-service-account')
|
||||
}
|
||||
return argv[1]
|
||||
}
|
||||
|
||||
export function classifyProductionCapacityDirector(state, capacityServiceAccount) {
|
||||
const currentIdentity = state.currentCapacityServiceAccount
|
||||
if (currentIdentity !== null && currentIdentity !== capacityServiceAccount) {
|
||||
throw new Error('director has an unexpected capacity identity')
|
||||
}
|
||||
const {
|
||||
baseCells,
|
||||
currentCells,
|
||||
capacityCellIds,
|
||||
targetCellId,
|
||||
targetHardCap
|
||||
} = state
|
||||
if (
|
||||
!Array.isArray(baseCells) ||
|
||||
!Array.isArray(currentCells) ||
|
||||
!Array.isArray(capacityCellIds) ||
|
||||
![600, 1000].includes(targetHardCap) ||
|
||||
new Set(capacityCellIds).size !== capacityCellIds.length ||
|
||||
!capacityCellIds.includes(targetCellId) ||
|
||||
baseCells.length !== currentCells.length ||
|
||||
new Set(baseCells.map((cell) => cell?.id)).size !== baseCells.length ||
|
||||
new Set(currentCells.map((cell) => cell?.id)).size !== currentCells.length
|
||||
) {
|
||||
throw new Error('director topology transition input is invalid')
|
||||
}
|
||||
const capacityCells = new Set(capacityCellIds)
|
||||
const normalizedCurrent = currentCells.map((current, index) => {
|
||||
const base = baseCells[index]
|
||||
if (
|
||||
typeof current?.id !== 'string' ||
|
||||
current.id !== base?.id
|
||||
) {
|
||||
throw new Error('director topology cell identity is invalid')
|
||||
}
|
||||
if (!capacityCells.has(current.id)) {
|
||||
if (!isDeepStrictEqual(current, base)) {
|
||||
throw new Error('director topology changed outside the capacity rollout')
|
||||
}
|
||||
return current
|
||||
}
|
||||
if (
|
||||
base.connectionHardCap !== 1000 ||
|
||||
base.connectionUnobservedBound !== 60 ||
|
||||
![600, 1000].includes(current.connectionHardCap) ||
|
||||
current.connectionUnobservedBound !== 60
|
||||
) {
|
||||
throw new Error('director capacity rollout state is invalid')
|
||||
}
|
||||
return {
|
||||
...current,
|
||||
connectionHardCap: base.connectionHardCap,
|
||||
connectionUnobservedBound: base.connectionUnobservedBound
|
||||
}
|
||||
})
|
||||
if (
|
||||
!isDeepStrictEqual(normalizedCurrent, baseCells) ||
|
||||
capacityCellIds.some((cellId) => !baseCells.some((cell) => cell.id === cellId))
|
||||
) {
|
||||
throw new Error('director topology is outside the reviewed capacity envelope')
|
||||
}
|
||||
const withTargetCap = (hardCap) => currentCells.map((cell) =>
|
||||
cell.id === targetCellId
|
||||
? { ...cell, connectionHardCap: hardCap, connectionUnobservedBound: 60 }
|
||||
: cell
|
||||
)
|
||||
const desiredCells = withTargetCap(targetHardCap)
|
||||
const predecessorCells = withTargetCap(targetHardCap === 600 ? 1000 : 600)
|
||||
const topologyPhase = isDeepStrictEqual(currentCells, desiredCells)
|
||||
? 'desired'
|
||||
: isDeepStrictEqual(currentCells, predecessorCells)
|
||||
? 'predecessor'
|
||||
: null
|
||||
if (!topologyPhase) throw new Error('director topology is not a reviewed transition state')
|
||||
return {
|
||||
topologyPhase,
|
||||
directorReady:
|
||||
topologyPhase === 'desired' && currentIdentity === capacityServiceAccount,
|
||||
desiredCells
|
||||
}
|
||||
}
|
||||
|
||||
export function main(argv = process.argv.slice(2)) {
|
||||
const capacityServiceAccount = parseArguments(argv)
|
||||
const state = JSON.parse(readFileSync(0, 'utf8'))
|
||||
process.stdout.write(`${JSON.stringify({
|
||||
event: 'relay_production_capacity_director_classified',
|
||||
...classifyProductionCapacityDirector(state, capacityServiceAccount)
|
||||
})}\n`)
|
||||
}
|
||||
|
||||
if (import.meta.url === pathToFileURL(process.argv[1]).href) {
|
||||
try {
|
||||
main()
|
||||
} catch (error) {
|
||||
process.stderr.write(`${error instanceof Error ? error.message : String(error)}\n`)
|
||||
process.exitCode = 1
|
||||
}
|
||||
}
|
||||
@@ -1,93 +0,0 @@
|
||||
import assert from 'node:assert/strict'
|
||||
import test from 'node:test'
|
||||
import {
|
||||
classifyProductionCapacityDirector
|
||||
} from './classify-relay-production-capacity-director.mjs'
|
||||
|
||||
const capacityServiceAccount =
|
||||
'orca-cloud-gha-relay-cap@onorca-cloud.iam.gserviceaccount.com'
|
||||
const capacityCellIds = ['production-gce-c25', 'production-gce-c26']
|
||||
const baseCells = [
|
||||
{ id: 'production-gce-c17', connectionHardCap: 600, connectionUnobservedBound: 60 },
|
||||
{ id: 'production-gce-c25', connectionHardCap: 1000, connectionUnobservedBound: 60 },
|
||||
{ id: 'production-gce-c26', connectionHardCap: 1000, connectionUnobservedBound: 60 }
|
||||
]
|
||||
const mixedCells = [
|
||||
baseCells[0],
|
||||
{ ...baseCells[1], connectionHardCap: 600 },
|
||||
baseCells[2]
|
||||
]
|
||||
|
||||
function state(overrides = {}) {
|
||||
return {
|
||||
baseCells,
|
||||
currentCells: mixedCells,
|
||||
capacityCellIds,
|
||||
targetCellId: 'production-gce-c25',
|
||||
targetHardCap: 1000,
|
||||
currentCapacityServiceAccount: capacityServiceAccount,
|
||||
...overrides
|
||||
}
|
||||
}
|
||||
|
||||
test('classifies one target while preserving completed rollout cells', () => {
|
||||
assert.deepEqual(
|
||||
classifyProductionCapacityDirector(state(), capacityServiceAccount),
|
||||
{
|
||||
topologyPhase: 'predecessor',
|
||||
directorReady: false,
|
||||
desiredCells: baseCells
|
||||
}
|
||||
)
|
||||
})
|
||||
|
||||
test('skips deployment only for exact topology and identity', () => {
|
||||
assert.deepEqual(
|
||||
classifyProductionCapacityDirector(state({ currentCells: baseCells }), capacityServiceAccount),
|
||||
{ topologyPhase: 'desired', directorReady: true, desiredCells: baseCells }
|
||||
)
|
||||
assert.deepEqual(
|
||||
classifyProductionCapacityDirector(state({
|
||||
currentCells: baseCells,
|
||||
targetHardCap: 600
|
||||
}), capacityServiceAccount),
|
||||
{
|
||||
topologyPhase: 'predecessor',
|
||||
directorReady: false,
|
||||
desiredCells: mixedCells
|
||||
}
|
||||
)
|
||||
})
|
||||
|
||||
test('rejects an unknown topology or capacity identity', () => {
|
||||
assert.throws(
|
||||
() => classifyProductionCapacityDirector(state({
|
||||
currentCells: [baseCells[0], { ...baseCells[1], connectionHardCap: 700 }, baseCells[2]]
|
||||
}), capacityServiceAccount),
|
||||
/rollout state is invalid/
|
||||
)
|
||||
assert.throws(
|
||||
() => classifyProductionCapacityDirector(state({
|
||||
currentCapacityServiceAccount: 'unexpected@onorca-cloud.iam.gserviceaccount.com'
|
||||
}), capacityServiceAccount),
|
||||
/unexpected capacity identity/
|
||||
)
|
||||
assert.throws(
|
||||
() => classifyProductionCapacityDirector(state({
|
||||
currentCells: [{ ...baseCells[0], connectionHardCap: 1000 }, mixedCells[1], mixedCells[2]]
|
||||
}), capacityServiceAccount),
|
||||
/outside the capacity rollout/
|
||||
)
|
||||
assert.throws(
|
||||
() => classifyProductionCapacityDirector(state({
|
||||
currentCells: [baseCells[0], { ...mixedCells[1], url: 'https://wrong.invalid' }, baseCells[2]]
|
||||
}), capacityServiceAccount),
|
||||
/outside the reviewed capacity envelope/
|
||||
)
|
||||
assert.throws(
|
||||
() => classifyProductionCapacityDirector(state({
|
||||
targetCellId: 'production-gce-c17'
|
||||
}), capacityServiceAccount),
|
||||
/transition input is invalid/
|
||||
)
|
||||
})
|
||||
@@ -1,47 +0,0 @@
|
||||
import { pathToFileURL } from 'node:url'
|
||||
|
||||
export function classifyStagingBootstrap({ c2Kind, c2Admission, c3Kind, c3Admission }) {
|
||||
for (const kind of [c2Kind, c3Kind]) {
|
||||
if (!['legacy', 'modern'].includes(kind)) throw new Error('bootstrap runtime kind is invalid')
|
||||
}
|
||||
for (const admission of [c2Admission, c3Admission]) {
|
||||
if (!['general', 'migration-only'].includes(admission)) {
|
||||
throw new Error('bootstrap admission is not recoverable')
|
||||
}
|
||||
}
|
||||
if (c2Kind === 'modern' && c3Kind === 'modern') return 'complete'
|
||||
if (c2Kind === 'modern') return 'roll-c3'
|
||||
if (c3Kind === 'modern') return 'roll-c2'
|
||||
if (c2Admission === 'general') return 'normalize-and-roll-both'
|
||||
if (c3Admission === 'general') return 'resume-c2-then-c3'
|
||||
throw new Error('bootstrap has no general fallback')
|
||||
}
|
||||
|
||||
function parseArguments(argv) {
|
||||
const values = {}
|
||||
for (let index = 0; index < argv.length; index += 2) {
|
||||
const key = argv[index]
|
||||
const value = argv[index + 1]
|
||||
if (!key?.startsWith('--') || value === undefined) throw new Error('invalid arguments')
|
||||
values[key.slice(2)] = value
|
||||
}
|
||||
return {
|
||||
c2Kind: values['c2-kind'],
|
||||
c2Admission: values['c2-admission'],
|
||||
c3Kind: values['c3-kind'],
|
||||
c3Admission: values['c3-admission']
|
||||
}
|
||||
}
|
||||
|
||||
export function main(argv = process.argv.slice(2)) {
|
||||
process.stdout.write(`${classifyStagingBootstrap(parseArguments(argv))}\n`)
|
||||
}
|
||||
|
||||
if (import.meta.url === pathToFileURL(process.argv[1]).href) {
|
||||
try {
|
||||
main()
|
||||
} catch (error) {
|
||||
process.stderr.write(`${error instanceof Error ? error.message : String(error)}\n`)
|
||||
process.exitCode = 1
|
||||
}
|
||||
}
|
||||
@@ -1,60 +0,0 @@
|
||||
import assert from 'node:assert/strict'
|
||||
import { test } from 'node:test'
|
||||
import { classifyStagingBootstrap } from './classify-relay-staging-bootstrap.mjs'
|
||||
|
||||
const state = (c2Kind, c2Admission, c3Kind, c3Admission) => ({
|
||||
c2Kind,
|
||||
c2Admission,
|
||||
c3Kind,
|
||||
c3Admission
|
||||
})
|
||||
|
||||
test('classifies the fresh legacy bootstrap', () => {
|
||||
assert.equal(
|
||||
classifyStagingBootstrap(state('legacy', 'general', 'legacy', 'general')),
|
||||
'normalize-and-roll-both'
|
||||
)
|
||||
})
|
||||
|
||||
test('retries normalization after a failed C3 restart', () => {
|
||||
assert.equal(
|
||||
classifyStagingBootstrap(state('legacy', 'general', 'legacy', 'migration-only')),
|
||||
'normalize-and-roll-both'
|
||||
)
|
||||
})
|
||||
|
||||
test('resumes after C2 isolation', () => {
|
||||
assert.equal(
|
||||
classifyStagingBootstrap(state('legacy', 'migration-only', 'legacy', 'general')),
|
||||
'resume-c2-then-c3'
|
||||
)
|
||||
})
|
||||
|
||||
test('resumes after C2 apply or a partial C3 transition', () => {
|
||||
for (const c2Admission of ['general', 'migration-only']) {
|
||||
for (const c3Admission of ['general', 'migration-only']) {
|
||||
assert.equal(
|
||||
classifyStagingBootstrap(state('modern', c2Admission, 'legacy', c3Admission)),
|
||||
'roll-c3'
|
||||
)
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
test('repairs an unexpected modern C3 before rolling legacy C2', () => {
|
||||
assert.equal(
|
||||
classifyStagingBootstrap(state('legacy', 'migration-only', 'modern', 'general')),
|
||||
'roll-c2'
|
||||
)
|
||||
})
|
||||
|
||||
test('accepts already complete modern cells and rejects no-fallback legacy state', () => {
|
||||
assert.equal(
|
||||
classifyStagingBootstrap(state('modern', 'migration-only', 'modern', 'general')),
|
||||
'complete'
|
||||
)
|
||||
assert.throws(
|
||||
() => classifyStagingBootstrap(state('legacy', 'migration-only', 'legacy', 'migration-only')),
|
||||
/no general fallback/
|
||||
)
|
||||
})
|
||||
@@ -1,314 +0,0 @@
|
||||
// Derives, from workflow and script content, which workflows roll out against the shared Cloud SQL
|
||||
// instance. Hand lists go stale silently; everything here is read back off disk.
|
||||
import { readFileSync, readdirSync } from 'node:fs'
|
||||
import {
|
||||
RELAY_WORKFLOW_DIRECTORY,
|
||||
RELAY_WORKFLOW_FILE_PREFIX,
|
||||
relayWorkflowFile
|
||||
} from './relay-repository.mjs'
|
||||
|
||||
export const WORKFLOW_ROOT = RELAY_WORKFLOW_DIRECTORY
|
||||
export const SCRIPT_ROOT = new URL('./', import.meta.url)
|
||||
|
||||
export const LEASE_ACTION = './.github/actions/cloud-sql-rollout-lease'
|
||||
export const PRODUCTION_LEASE = {
|
||||
bucket: 'onorca-cloud-terraform-state',
|
||||
object: 'terraform/state/cloud-sql-rollout/production.lock'
|
||||
}
|
||||
export const STAGING_LEASE = {
|
||||
bucket: 'onorca-cloud-staging-terraform-state',
|
||||
object: 'terraform/state/cloud-sql-rollout/staging.lock'
|
||||
}
|
||||
|
||||
export const PRODUCTION_GROUP = 'production-cloud-sql-rollout'
|
||||
export const STAGING_GROUP = 'relay-staging-mutation'
|
||||
export const SELECTABLE_GROUP =
|
||||
"${{ inputs.environment == 'production' && 'production-cloud-sql-rollout' || 'relay-staging-mutation' }}"
|
||||
// Prove Relay Asia Staging's load mode drives traffic only and must not hold the staging lock.
|
||||
export const STAGING_OR_LOAD_GROUP =
|
||||
"${{ inputs.mode == 'load' && 'relay-staging-load' || 'relay-staging-mutation' }}"
|
||||
|
||||
const selectable = (production, staging) =>
|
||||
`\${{ inputs.environment == 'production' && '${production}' || '${staging}' }}`
|
||||
|
||||
export const SELECTABLE_LEASE = {
|
||||
bucket: selectable(PRODUCTION_LEASE.bucket, STAGING_LEASE.bucket),
|
||||
object: selectable(PRODUCTION_LEASE.object, STAGING_LEASE.object)
|
||||
}
|
||||
|
||||
export const LOCK_GROUPS = new Set([
|
||||
PRODUCTION_GROUP,
|
||||
STAGING_GROUP,
|
||||
SELECTABLE_GROUP,
|
||||
STAGING_OR_LOAD_GROUP
|
||||
])
|
||||
|
||||
export function readWorkflow(file) {
|
||||
return readFileSync(new URL(file, WORKFLOW_ROOT), 'utf8')
|
||||
}
|
||||
|
||||
export function workflowFiles() {
|
||||
return readdirSync(WORKFLOW_ROOT)
|
||||
.filter((name) => name.endsWith('.yml') && name.startsWith(RELAY_WORKFLOW_FILE_PREFIX))
|
||||
.sort()
|
||||
}
|
||||
|
||||
// --- YAML-shaped readers (line based; the workflows are hand-written and uniformly indented) ---
|
||||
|
||||
function indentOf(line) {
|
||||
return line.length - line.trimStart().length
|
||||
}
|
||||
|
||||
function blockAfter(lines, index) {
|
||||
const base = indentOf(lines[index])
|
||||
const body = []
|
||||
for (let i = index + 1; i < lines.length; i += 1) {
|
||||
if (lines[i].trim() === '') {
|
||||
body.push(lines[i])
|
||||
continue
|
||||
}
|
||||
if (indentOf(lines[i]) <= base) break
|
||||
body.push(lines[i])
|
||||
}
|
||||
return body
|
||||
}
|
||||
|
||||
export function concurrencyBlocks(text) {
|
||||
const lines = text.split('\n')
|
||||
const blocks = []
|
||||
lines.forEach((line, index) => {
|
||||
if (line.trim() !== 'concurrency:') return
|
||||
const body = blockAfter(lines, index)
|
||||
blocks.push({
|
||||
group: body.find((l) => l.trim().startsWith('group:'))?.trim().slice('group:'.length).trim(),
|
||||
cancelInProgress: body
|
||||
.find((l) => l.trim().startsWith('cancel-in-progress:'))
|
||||
?.trim()
|
||||
.slice('cancel-in-progress:'.length)
|
||||
.trim()
|
||||
})
|
||||
})
|
||||
return blocks
|
||||
}
|
||||
|
||||
export function jobs(text) {
|
||||
const lines = text.split('\n')
|
||||
const start = lines.findIndex((line) => line === 'jobs:')
|
||||
if (start === -1) return []
|
||||
const found = []
|
||||
for (let i = start + 1; i < lines.length; i += 1) {
|
||||
const match = /^ {2}([A-Za-z0-9_-]+):\s*$/.exec(lines[i])
|
||||
if (!match) continue
|
||||
found.push({ id: match[1], start: i, body: blockAfter(lines, i) })
|
||||
}
|
||||
return found.map((job) => ({ ...job, text: job.body.join('\n') }))
|
||||
}
|
||||
|
||||
function scalarField(jobText, key) {
|
||||
const lines = jobText.split('\n')
|
||||
const index = lines.findIndex((line) => /^ {4}[A-Za-z-]+:/.test(line) && line.trim().startsWith(`${key}:`))
|
||||
if (index === -1) return undefined
|
||||
const inline = lines[index].trim().slice(`${key}:`.length).trim()
|
||||
if (inline !== '' && inline !== '>-' && inline !== '|') return inline
|
||||
return blockAfter(lines, index).join(' ').replace(/\s+/g, ' ').trim()
|
||||
}
|
||||
|
||||
export function jobNeeds(jobText) {
|
||||
const raw = scalarField(jobText, 'needs')
|
||||
if (!raw) return []
|
||||
return raw
|
||||
.replace(/^\[|\]$/g, '')
|
||||
.split(/[,\n]|\s+-\s+/)
|
||||
.map((entry) => entry.replace(/^-/, '').trim())
|
||||
.filter(Boolean)
|
||||
}
|
||||
|
||||
export function jobIf(jobText) {
|
||||
return scalarField(jobText, 'if') ?? ''
|
||||
}
|
||||
|
||||
export function leaseSteps(text) {
|
||||
const lines = text.split('\n')
|
||||
const steps = []
|
||||
lines.forEach((line, index) => {
|
||||
if (line.trim() !== `- uses: ${LEASE_ACTION}`) return
|
||||
const body = blockAfter(lines, index)
|
||||
const read = (key) =>
|
||||
body.find((l) => l.trim().startsWith(`${key}:`))?.trim().slice(`${key}:`.length).trim()
|
||||
steps.push({
|
||||
line: index + 1,
|
||||
bucket: read('bucket'),
|
||||
object: read('object'),
|
||||
release: read('release')
|
||||
})
|
||||
})
|
||||
return steps
|
||||
}
|
||||
|
||||
export function leaseStepsByJob(file) {
|
||||
const text = readWorkflow(file)
|
||||
const steps = leaseSteps(text)
|
||||
return jobs(text).map((job) => ({
|
||||
id: job.id,
|
||||
steps: steps.filter((step) => step.line > job.start + 1 && step.line <= job.start + 1 + job.body.length)
|
||||
}))
|
||||
}
|
||||
|
||||
// --- trigger and reusable-call graph ---
|
||||
|
||||
export function triggers(text) {
|
||||
const lines = text.split('\n')
|
||||
const index = lines.findIndex((line) => line === 'on:')
|
||||
if (index === -1) return []
|
||||
return blockAfter(lines, index)
|
||||
.map((line) => /^ {2}([a-z_]+):/.exec(line)?.[1])
|
||||
.filter(Boolean)
|
||||
}
|
||||
|
||||
export function isEntrypoint(text) {
|
||||
return triggers(text).some((trigger) => trigger !== 'workflow_call')
|
||||
}
|
||||
|
||||
export function reusableCalls(text) {
|
||||
const counts = new Map()
|
||||
for (const match of text.matchAll(/uses: \.\/\.github\/workflows\/([A-Za-z0-9._-]+\.yml)/g)) {
|
||||
counts.set(match[1], (counts.get(match[1]) ?? 0) + 1)
|
||||
}
|
||||
return counts
|
||||
}
|
||||
|
||||
export function entrypointsFor(file, seen = new Set()) {
|
||||
if (seen.has(file)) return new Set()
|
||||
seen.add(file)
|
||||
if (isEntrypoint(readWorkflow(file))) return new Set([file])
|
||||
const reached = new Set()
|
||||
for (const candidate of workflowFiles()) {
|
||||
if (candidate === file) continue
|
||||
if (!reusableCalls(readWorkflow(candidate)).has(file)) continue
|
||||
for (const entry of entrypointsFor(candidate, seen)) reached.add(entry)
|
||||
}
|
||||
return reached
|
||||
}
|
||||
|
||||
// --- what counts as a Cloud SQL connection-budget rollout ---
|
||||
|
||||
const COMMAND_PREFIX = /^(?:-\s+)?(?:run:\s*)?(?:[a-z_]+\s*=\s*"?\$\(\s*)?(?:if\s+|then\s+|else\s+|&&\s+|\|\|\s+|!\s+)*/
|
||||
|
||||
function commandLines(text) {
|
||||
return text.split('\n').map((line) => line.trim().replace(COMMAND_PREFIX, ''))
|
||||
}
|
||||
|
||||
export function appliesTerraform(text) {
|
||||
return commandLines(text).some((line) => /^terraform\b.*\bapply\b/.test(line))
|
||||
}
|
||||
|
||||
export function runsCloudRunMutation(text) {
|
||||
return commandLines(text).some((line) =>
|
||||
/^gcloud run (?:deploy\b|services (?:update|replace)\b|jobs (?:update|deploy)\b)/.test(line)
|
||||
)
|
||||
}
|
||||
|
||||
// Scripts that mint a Cloud Run revision, plus every script that re-exports one of them.
|
||||
export function revisionMintingScripts() {
|
||||
const self = new URL(import.meta.url).pathname.split('/').pop()
|
||||
const names = readdirSync(SCRIPT_ROOT).filter(
|
||||
(name) => name.endsWith('.mjs') && !name.endsWith('.test.mjs') && name !== self
|
||||
)
|
||||
const source = new Map(
|
||||
names.map((name) => [name, readFileSync(new URL(name, SCRIPT_ROOT), 'utf8')])
|
||||
)
|
||||
const minting = new Set(
|
||||
names.filter((name) => {
|
||||
const text = source.get(name)
|
||||
return (
|
||||
text.includes("'--no-traffic'") ||
|
||||
/'run',\s*'services',\s*'update'/.test(text) ||
|
||||
/'run',\s*'deploy'/.test(text)
|
||||
)
|
||||
})
|
||||
)
|
||||
for (let changed = true; changed; ) {
|
||||
changed = false
|
||||
for (const name of names) {
|
||||
if (minting.has(name)) continue
|
||||
const imports = [...source.get(name).matchAll(/from '\.\/([A-Za-z0-9._-]+\.mjs)'/g)].map(
|
||||
(match) => match[1]
|
||||
)
|
||||
if (!imports.some((imported) => minting.has(imported))) continue
|
||||
minting.add(name)
|
||||
changed = true
|
||||
}
|
||||
}
|
||||
return minting
|
||||
}
|
||||
|
||||
export function mutatesSharedInstance(text, minters = revisionMintingScripts()) {
|
||||
if (appliesTerraform(text)) return 'terraform apply against the reviewed relay cell templates'
|
||||
if (runsCloudRunMutation(text)) return 'gcloud mints or replaces a Cloud Run revision'
|
||||
for (const script of minters) {
|
||||
if (text.includes(`dev/scripts/${script}`)) return `runs ${script}, which mints a Cloud Run revision`
|
||||
}
|
||||
return undefined
|
||||
}
|
||||
|
||||
// --- the declared contract ---
|
||||
|
||||
const production = (extra = {}) => ({ env: 'production', group: PRODUCTION_GROUP, ...extra })
|
||||
const staging = (extra = {}) => ({ env: 'staging', group: STAGING_GROUP, ...extra })
|
||||
const eitherEnvironment = () => ({ env: 'selectable', group: SELECTABLE_GROUP })
|
||||
|
||||
// Keys are workflow filenames, which the public copy prefixes; the prefix lives in one place.
|
||||
const named = (entries) =>
|
||||
Object.fromEntries(
|
||||
entries.map(([file, entry]) => [
|
||||
relayWorkflowFile(file),
|
||||
entry.leaseFiles
|
||||
? { ...entry, leaseFiles: entry.leaseFiles.map((member) => relayWorkflowFile(member)) }
|
||||
: entry
|
||||
])
|
||||
)
|
||||
|
||||
export const LEASED_WORKFLOWS = named([
|
||||
['deploy-relay-fence-broker.yml', production()],
|
||||
['deploy-relay-production.yml', production()],
|
||||
['deploy-relay-production-director.yml', production()],
|
||||
['deploy-relay-production-multi-target.yml', production()],
|
||||
[
|
||||
'deploy-relay-production-capacity.yml',
|
||||
production({
|
||||
leaseFiles: ['deploy-relay-production-capacity-job.yml'],
|
||||
reentrant: true
|
||||
})
|
||||
],
|
||||
[
|
||||
'deploy-relay-production-same-cap.yml',
|
||||
production({
|
||||
leaseFiles: ['deploy-relay-production-same-cap-job.yml'],
|
||||
reentrant: true
|
||||
})
|
||||
],
|
||||
[
|
||||
'operate-relay-production-rehome.yml',
|
||||
production({ leaseFiles: ['operate-relay-production-rehome-job.yml'] })
|
||||
],
|
||||
['deploy-relay-asia-topology.yml', eitherEnvironment()],
|
||||
['operate-relay-asia-admission.yml', eitherEnvironment()],
|
||||
['deploy-relay-staging.yml', staging()],
|
||||
['deploy-relay-staging-gce-candidate.yml', staging()],
|
||||
['bootstrap-relay-staging-capacity.yml', staging()],
|
||||
['power-relay-staging.yml', staging()],
|
||||
['prove-relay-asia-staging.yml', staging({ group: STAGING_OR_LOAD_GROUP })],
|
||||
[
|
||||
'prove-relay-staging-capacity.yml',
|
||||
staging({ exclusiveBy: "inputs.mode == 'refresh-asia-c4-image'" })
|
||||
],
|
||||
['recover-relay-staging-c4-image.yml', staging()]
|
||||
])
|
||||
|
||||
export const NOT_A_CLOUD_SQL_CANDIDATE = named([
|
||||
['push-deploy.yml', 'Push uses dedicated SQL and its own production-push-rollout group and durable push-rollout lease.'],
|
||||
[
|
||||
'monitor-relay-production.yml',
|
||||
'Read-only. Its identity holds monitoring, logging, Cloud SQL and compute viewer roles only, and it runs `gcloud sql instances describe`, never a mutation. It consumes no connection budget, so the durable lease would only let monitoring block a rollout and a rollout block monitoring.'
|
||||
]
|
||||
])
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -1,876 +0,0 @@
|
||||
import { readFileSync } from 'node:fs'
|
||||
import { spawnSync } from 'node:child_process'
|
||||
import { pathToFileURL } from 'node:url'
|
||||
import {
|
||||
inspectAdmissionSelector,
|
||||
selectorCellState,
|
||||
transitionAdmissionSelector
|
||||
} from './relay-admission-selector.mjs'
|
||||
import { isReserveCellRefusal, reserveCellRollRefusal } from './relay-reserve-cell-refusal.mjs'
|
||||
|
||||
const DEFAULT_POLL_INTERVAL_MS = 5_000
|
||||
const DEFAULT_TIMEOUT_MS = 14 * 60 * 1_000
|
||||
const ADMIN_RETRY_ATTEMPTS = 3
|
||||
const ADMIN_RETRY_BASE_MS = 250
|
||||
const CONNECTION_CONTROL_REBIND_RESERVE = 100
|
||||
const SUPPORTED_CONNECTION_HARD_CAPS = new Set([600, 1_000, 3_000])
|
||||
const RETRYABLE_ADMIN_PATHS = new Set([
|
||||
'/v1/admin/runtime-status',
|
||||
'/v1/admin/cell-status',
|
||||
'/v1/admin/evacuation-capacity',
|
||||
'/v1/admin/evacuation-status'
|
||||
])
|
||||
|
||||
function canonicalOrigin(value, name) {
|
||||
const url = new URL(value)
|
||||
if (url.protocol !== 'https:' || url.origin !== value || url.pathname !== '/') {
|
||||
throw new Error(`${name} must be a canonical HTTPS origin`)
|
||||
}
|
||||
return value
|
||||
}
|
||||
|
||||
function adminAudience(value) {
|
||||
const url = new URL(value)
|
||||
if (
|
||||
url.protocol !== 'https:' ||
|
||||
url.pathname !== '/v1/admin/drain' ||
|
||||
url.search ||
|
||||
url.hash ||
|
||||
url.toString() !== value
|
||||
) {
|
||||
throw new Error('--admin-audience must be the canonical HTTPS director drain URL')
|
||||
}
|
||||
return value
|
||||
}
|
||||
|
||||
function positiveInteger(value, name, maximum = Number.MAX_SAFE_INTEGER) {
|
||||
const parsed = Number(value)
|
||||
if (!Number.isInteger(parsed) || parsed <= 0 || parsed > maximum) {
|
||||
throw new Error(`${name} must be a positive integer`)
|
||||
}
|
||||
return parsed
|
||||
}
|
||||
|
||||
function nonnegativeInteger(value, name) {
|
||||
const parsed = Number(value)
|
||||
if (!Number.isInteger(parsed) || parsed < 0) {
|
||||
throw new Error(`${name} must be a nonnegative integer`)
|
||||
}
|
||||
return parsed
|
||||
}
|
||||
|
||||
export function parseArguments(argv) {
|
||||
const values = {}
|
||||
for (let index = 0; index < argv.length; index += 2) {
|
||||
const key = argv[index]
|
||||
const value = argv[index + 1]
|
||||
if (!key?.startsWith('--') || value === undefined) throw new Error(`invalid argument ${key ?? ''}`)
|
||||
values[key.slice(2)] = value
|
||||
}
|
||||
for (const key of [
|
||||
'project',
|
||||
'director-origin',
|
||||
'admin-audience',
|
||||
'topology-file',
|
||||
'source-cell-id',
|
||||
'target-cell-id',
|
||||
'runtime-service-account',
|
||||
'mode'
|
||||
]) {
|
||||
if (!values[key]) throw new Error(`missing --${key}`)
|
||||
}
|
||||
if (
|
||||
![
|
||||
'audit',
|
||||
'preflight',
|
||||
'recover-forward',
|
||||
'continue-evacuation',
|
||||
'disable-cell',
|
||||
'execute',
|
||||
'reset-empty-candidate',
|
||||
'enable-empty-cell'
|
||||
].includes(values.mode)
|
||||
) {
|
||||
throw new Error(
|
||||
'--mode must be audit, preflight, recover-forward, continue-evacuation, disable-cell, execute, reset-empty-candidate, or enable-empty-cell'
|
||||
)
|
||||
}
|
||||
return {
|
||||
project: values.project,
|
||||
directorOrigin: canonicalOrigin(values['director-origin'], '--director-origin'),
|
||||
adminAudience: adminAudience(values['admin-audience']),
|
||||
topologyFile: values['topology-file'],
|
||||
sourceCellId: values['source-cell-id'],
|
||||
targetCellId: values['target-cell-id'],
|
||||
runtimeServiceAccount: values['runtime-service-account'],
|
||||
mode: values.mode,
|
||||
batchSize: positiveInteger(values['batch-size'] ?? 100, '--batch-size', 100),
|
||||
drainGraceMs: positiveInteger(
|
||||
values['drain-grace-ms'] ?? 120_000,
|
||||
'--drain-grace-ms',
|
||||
60 * 60 * 1_000
|
||||
),
|
||||
pollIntervalMs: positiveInteger(
|
||||
values['poll-interval-ms'] ?? DEFAULT_POLL_INTERVAL_MS,
|
||||
'--poll-interval-ms',
|
||||
60_000
|
||||
),
|
||||
timeoutMs: positiveInteger(
|
||||
values['timeout-ms'] ?? DEFAULT_TIMEOUT_MS,
|
||||
'--timeout-ms',
|
||||
60 * 60 * 1_000
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
export function deployment(value, cellId) {
|
||||
if (!value || typeof value !== 'object') throw new Error(`missing topology for ${cellId}`)
|
||||
const expected = {
|
||||
cellId,
|
||||
origin: canonicalOrigin(value.origin, `${cellId} origin`),
|
||||
region: String(value.region ?? 'us-central1'),
|
||||
zone: String(value.zone ?? ''),
|
||||
migName: String(value.mig_name ?? ''),
|
||||
instanceGroup: String(value.instance_group ?? ''),
|
||||
backendName: String(value.backend_name ?? ''),
|
||||
backendId: String(value.backend_id ?? ''),
|
||||
urlMapName: String(value.url_map_name ?? ''),
|
||||
generationIdentity: String(value.generation_identity ?? ''),
|
||||
image: String(value.image ?? ''),
|
||||
imageDigest: String(value.image ?? '').split('@')[1] ?? '',
|
||||
capacityRequests: positiveInteger(value.capacity_requests, `${cellId} capacity`),
|
||||
databasePoolMax: positiveInteger(
|
||||
value.database_pool_max ?? 10,
|
||||
`${cellId} database pool maximum`,
|
||||
100
|
||||
),
|
||||
connectionHardCap:
|
||||
value.connection_hard_cap === null || value.connection_hard_cap === undefined
|
||||
? undefined
|
||||
: positiveInteger(value.connection_hard_cap, `${cellId} connection hard cap`),
|
||||
connectionUnobservedBound:
|
||||
value.connection_unobserved_bound === null ||
|
||||
value.connection_unobserved_bound === undefined
|
||||
? undefined
|
||||
: nonnegativeInteger(
|
||||
value.connection_unobserved_bound,
|
||||
`${cellId} unobserved connection bound`
|
||||
),
|
||||
initiallyEnabled: value.initially_enabled,
|
||||
fenced: value.fenced,
|
||||
desiredTargetSize: value.desired_target_size
|
||||
}
|
||||
if (!/^[a-z0-9-]+$/.test(expected.zone)) throw new Error(`${cellId} has an invalid zone`)
|
||||
if (!['us-central1', 'asia-east2'].includes(expected.region) || !expected.zone.startsWith(`${expected.region}-`)) {
|
||||
throw new Error(`${cellId} has an invalid region`)
|
||||
}
|
||||
for (const [name, resource] of [
|
||||
['MIG', expected.migName],
|
||||
['instance group', expected.instanceGroup],
|
||||
['backend', expected.backendName],
|
||||
['backend ID', expected.backendId]
|
||||
]) {
|
||||
if (!resource) throw new Error(`${cellId} has no ${name}`)
|
||||
}
|
||||
if (!/^[a-z0-9.-]+\/[a-z0-9._/-]+@sha256:[a-f0-9]{64}$/.test(expected.image)) {
|
||||
throw new Error(`${cellId} image is not digest-pinned`)
|
||||
}
|
||||
if (typeof expected.initiallyEnabled !== 'boolean') {
|
||||
throw new Error(`${cellId} has no initial admission state`)
|
||||
}
|
||||
if (
|
||||
(expected.connectionHardCap === undefined) !==
|
||||
(expected.connectionUnobservedBound === undefined) ||
|
||||
(expected.connectionHardCap !== undefined &&
|
||||
(!SUPPORTED_CONNECTION_HARD_CAPS.has(expected.connectionHardCap) ||
|
||||
expected.connectionUnobservedBound >=
|
||||
expected.connectionHardCap - CONNECTION_CONTROL_REBIND_RESERVE))
|
||||
) {
|
||||
throw new Error(`${cellId} has invalid connection capacity`)
|
||||
}
|
||||
return expected
|
||||
}
|
||||
|
||||
export function assertDeploymentConnectionCapacity(expected, runtime, director) {
|
||||
if (expected.connectionHardCap === undefined) {
|
||||
if (runtime !== null || director !== null) {
|
||||
throw new Error(`${expected.cellId} connection capacity differs from Terraform`)
|
||||
}
|
||||
return
|
||||
}
|
||||
const hardCap = expected.connectionHardCap
|
||||
const unobservedBound = expected.connectionUnobservedBound
|
||||
const ordinaryConnectionLimit = hardCap - CONNECTION_CONTROL_REBIND_RESERVE
|
||||
const normalAdmissionPause = ordinaryConnectionLimit - unobservedBound
|
||||
const matches = (capacity) =>
|
||||
capacity?.hardCap === hardCap &&
|
||||
capacity.controlRebindReserve === CONNECTION_CONTROL_REBIND_RESERVE &&
|
||||
capacity.ordinaryConnectionLimit === ordinaryConnectionLimit &&
|
||||
capacity.unobservedBound === unobservedBound &&
|
||||
capacity.normalAdmissionPause === normalAdmissionPause
|
||||
if (!matches(runtime) || !matches(director) || director.heartbeatFresh !== true) {
|
||||
throw new Error(`${expected.cellId} connection capacity differs from Terraform`)
|
||||
}
|
||||
}
|
||||
|
||||
export function selectDeployments(topology, sourceCellId, targetCellId) {
|
||||
if (sourceCellId === targetCellId) throw new Error('source and target cell IDs must differ')
|
||||
const source = deployment(topology[sourceCellId], sourceCellId)
|
||||
const target = deployment(topology[targetCellId], targetCellId)
|
||||
for (const key of ['origin', 'migName', 'instanceGroup', 'backendName', 'backendId']) {
|
||||
if (source[key] === target[key]) throw new Error(`source and target ${key} overlap`)
|
||||
}
|
||||
if (target.initiallyEnabled) throw new Error('candidate must be declared initially disabled')
|
||||
return { source, target }
|
||||
}
|
||||
|
||||
export function validateMig(mig, instances, expected) {
|
||||
if (Number(mig.targetSize) !== 1) throw new Error(`${expected.cellId} MIG is not fixed-one`)
|
||||
const policy = mig.updatePolicy ?? {}
|
||||
if (
|
||||
policy.replacementMethod !== 'RECREATE' ||
|
||||
Number(policy.maxSurge?.fixed ?? policy.maxSurge) !== 0 ||
|
||||
Number(policy.maxUnavailable?.fixed ?? policy.maxUnavailable) !== 1
|
||||
) {
|
||||
throw new Error(`${expected.cellId} MIG replacement policy is unsafe`)
|
||||
}
|
||||
const serving = instances.filter(
|
||||
(entry) => entry.instanceStatus === 'RUNNING' && entry.currentAction === 'NONE'
|
||||
)
|
||||
if (instances.length !== 1 || serving.length !== 1) {
|
||||
throw new Error(`${expected.cellId} MIG must have one running endpoint`)
|
||||
}
|
||||
return serving[0].instance.split('/').at(-1)
|
||||
}
|
||||
|
||||
export function validateInstance(instance, expected, runtimeServiceAccount) {
|
||||
const publicConfigs = (instance.networkInterfaces ?? []).flatMap(
|
||||
(network) => network.accessConfigs ?? []
|
||||
)
|
||||
if (publicConfigs.length !== 0) throw new Error(`${expected.cellId} instance has a public IP`)
|
||||
const serviceAccounts = (instance.serviceAccounts ?? []).map((entry) => entry.email)
|
||||
if (serviceAccounts.length !== 1 || serviceAccounts[0] !== runtimeServiceAccount) {
|
||||
throw new Error(`${expected.cellId} runtime service account mismatch`)
|
||||
}
|
||||
}
|
||||
|
||||
export function validateBackend(backend, expected) {
|
||||
if (
|
||||
backend.protocol !== 'HTTP' ||
|
||||
Number(backend.timeoutSec) !== 86_400 ||
|
||||
(backend.backends ?? []).length !== 1 ||
|
||||
backend.backends[0].group !== expected.instanceGroup
|
||||
) {
|
||||
throw new Error(`${expected.cellId} backend topology mismatch`)
|
||||
}
|
||||
}
|
||||
|
||||
export function defaultCommandJson(args) {
|
||||
const result = spawnSync('gcloud', args, { encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe'] })
|
||||
if (result.status !== 0) {
|
||||
throw new Error(`gcloud ${args.slice(0, 4).join(' ')} failed: ${result.stderr.trim()}`)
|
||||
}
|
||||
return JSON.parse(result.stdout)
|
||||
}
|
||||
|
||||
export function suppliedAdminIdentityToken(environment = process.env) {
|
||||
const token = environment.ORCA_RELAY_ADMIN_ID_TOKEN
|
||||
if (token === undefined) return null
|
||||
return validatedIdentityToken(token, 'admin')
|
||||
}
|
||||
|
||||
export function suppliedFenceMutationIdentityToken(environment = process.env) {
|
||||
const token = environment.ORCA_RELAY_FENCE_MUTATION_ID_TOKEN
|
||||
if (token === undefined) return null
|
||||
return validatedIdentityToken(token, 'fence mutation')
|
||||
}
|
||||
|
||||
function validatedIdentityToken(token, label) {
|
||||
// WIF supplies a masked Google ID token because external-account gcloud cannot mint one directly.
|
||||
if (token.length > 8_192 || !/^[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+$/.test(token)) {
|
||||
throw new Error(`invalid supplied ${label} identity token`)
|
||||
}
|
||||
return token
|
||||
}
|
||||
|
||||
export function defaultIdentityToken(audience) {
|
||||
const supplied = suppliedAdminIdentityToken()
|
||||
if (supplied !== null) return supplied
|
||||
const result = spawnSync(
|
||||
'gcloud',
|
||||
['auth', 'print-identity-token', `--audiences=${audience}`],
|
||||
{ encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe'] }
|
||||
)
|
||||
if (result.status !== 0) throw new Error('gcloud identity-token command failed')
|
||||
return result.stdout.trim()
|
||||
}
|
||||
|
||||
async function responseJson(response, label, cellId) {
|
||||
const body = await response.json().catch(() => ({ error: `http_${response.status}` }))
|
||||
if (isReserveCellRefusal(response.status, body)) throw reserveCellRollRefusal(cellId)
|
||||
if (!response.ok) throw new Error(`${label} failed: ${body.error ?? response.status}`)
|
||||
return body
|
||||
}
|
||||
|
||||
export function createAdminPost(config, deps, token) {
|
||||
return async (origin, path, body) => {
|
||||
const requestToken = typeof token === 'function' ? token(path) : token
|
||||
for (let attempt = 1; attempt <= ADMIN_RETRY_ATTEMPTS; attempt++) {
|
||||
let response
|
||||
try {
|
||||
response = await deps.fetch(`${origin}${path}`, {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
authorization: `Bearer ${requestToken}`,
|
||||
'content-type': 'application/json'
|
||||
},
|
||||
body: JSON.stringify(body),
|
||||
signal: AbortSignal.timeout(30_000)
|
||||
})
|
||||
} catch (error) {
|
||||
if (!RETRYABLE_ADMIN_PATHS.has(path) || attempt === ADMIN_RETRY_ATTEMPTS) throw error
|
||||
deps.emit({ event: 'candidate_admin_retry', path, attempt, reason: 'transport' })
|
||||
await deps.wait(deps.random() * ADMIN_RETRY_BASE_MS * 2 ** (attempt - 1))
|
||||
continue
|
||||
}
|
||||
if (
|
||||
RETRYABLE_ADMIN_PATHS.has(path) &&
|
||||
([502, 503, 504].includes(response.status) ||
|
||||
(path === '/v1/admin/evacuation-status' && response.status === 500)) &&
|
||||
attempt < ADMIN_RETRY_ATTEMPTS
|
||||
) {
|
||||
// These endpoints are read-only or transactionally idempotent, so a
|
||||
// lost response may be retried without widening deployment authority.
|
||||
deps.emit({
|
||||
event: 'candidate_admin_retry',
|
||||
path,
|
||||
attempt,
|
||||
reason: `http_${response.status}`
|
||||
})
|
||||
await response.arrayBuffer().catch(() => undefined)
|
||||
await deps.wait(deps.random() * ADMIN_RETRY_BASE_MS * 2 ** (attempt - 1))
|
||||
continue
|
||||
}
|
||||
// The director names no cell, so a two-cell request names both.
|
||||
const cells =
|
||||
body?.cellId ?? [body?.sourceCellId, body?.targetCellId].filter(Boolean).join(' or ')
|
||||
return await responseJson(response, path, cells || null)
|
||||
}
|
||||
throw new Error(`${path} retry attempts exhausted`)
|
||||
}
|
||||
}
|
||||
|
||||
async function checkHttp(deps, origin, path) {
|
||||
const response = await deps.fetch(`${origin}${path}`, { signal: AbortSignal.timeout(15_000) })
|
||||
const body = await response.json().catch(() => ({}))
|
||||
if (!response.ok || body.ok !== true) throw new Error(`${origin}${path} is unavailable`)
|
||||
}
|
||||
|
||||
export async function inspectCell(config, deps, adminPost, expected) {
|
||||
const common = ['--project', config.project, '--zone', expected.zone, '--format=json']
|
||||
const mig = deps.commandJson([
|
||||
'compute',
|
||||
'instance-groups',
|
||||
'managed',
|
||||
'describe',
|
||||
expected.migName,
|
||||
...common
|
||||
])
|
||||
const instances = deps.commandJson([
|
||||
'compute',
|
||||
'instance-groups',
|
||||
'managed',
|
||||
'list-instances',
|
||||
expected.migName,
|
||||
...common
|
||||
])
|
||||
const instanceName = validateMig(mig, instances, expected)
|
||||
const instance = deps.commandJson([
|
||||
'compute',
|
||||
'instances',
|
||||
'describe',
|
||||
instanceName,
|
||||
...common
|
||||
])
|
||||
validateInstance(instance, expected, config.runtimeServiceAccount)
|
||||
const backend = deps.commandJson([
|
||||
'compute',
|
||||
'backend-services',
|
||||
'describe',
|
||||
expected.backendName,
|
||||
'--global',
|
||||
'--project',
|
||||
config.project,
|
||||
'--format=json'
|
||||
])
|
||||
validateBackend(backend, expected)
|
||||
await checkHttp(deps, expected.origin, '/health')
|
||||
await checkHttp(deps, expected.origin, '/ready')
|
||||
const runtime = await adminPost(expected.origin, '/v1/admin/runtime-status', { v: 1 })
|
||||
if (
|
||||
runtime.role !== 'cell' ||
|
||||
runtime.cellId !== expected.cellId ||
|
||||
runtime.cellUrl !== expected.origin ||
|
||||
(runtime.region ?? 'us-central1') !== expected.region ||
|
||||
runtime.imageDigest !== expected.imageDigest
|
||||
) {
|
||||
throw new Error(`${expected.cellId} served runtime does not match Terraform topology`)
|
||||
}
|
||||
const status = await adminPost(config.directorOrigin, '/v1/admin/cell-status', {
|
||||
v: 1,
|
||||
cellId: expected.cellId
|
||||
})
|
||||
if (
|
||||
status.status?.cellUrl !== expected.origin ||
|
||||
(status.status?.region ?? 'us-central1') !== expected.region ||
|
||||
status.status?.runtime?.cellUrl !== expected.origin ||
|
||||
status.status?.runtime?.ready !== true ||
|
||||
status.status?.runtime?.heartbeatFresh !== true
|
||||
) {
|
||||
throw new Error(`${expected.cellId} has no fresh ready authenticated heartbeat`)
|
||||
}
|
||||
assertDeploymentConnectionCapacity(
|
||||
expected,
|
||||
runtime.connectionCapacity ?? null,
|
||||
status.status.connectionCapacity ?? null
|
||||
)
|
||||
return {
|
||||
...status.status,
|
||||
draining: runtime.draining === true,
|
||||
process: runtime.runtime ?? null,
|
||||
runtimeConnectionCapacity: runtime.connectionCapacity ?? null
|
||||
}
|
||||
}
|
||||
|
||||
async function waitForMigration(config, deps, adminPost, completeReady) {
|
||||
const deadline = deps.now() + config.timeoutMs
|
||||
while (deps.now() < deadline) {
|
||||
const status = await adminPost(config.directorOrigin, '/v1/admin/evacuation-status', {
|
||||
v: 1,
|
||||
sourceCellId: config.sourceCellId,
|
||||
targetCellId: config.targetCellId,
|
||||
completeReady
|
||||
})
|
||||
deps.emit({ event: completeReady ? 'migration_completion' : 'migration_registration', ...status })
|
||||
if (completeReady ? status.inProgress === 0 : status.inProgress === status.targetRegistered) {
|
||||
return status
|
||||
}
|
||||
if (
|
||||
completeReady &&
|
||||
status.targetRegistered === status.inProgress &&
|
||||
status.registeredSourceActive === 0 &&
|
||||
status.registeredCompletable === 0 &&
|
||||
status.registeredTargetInactive === status.inProgress
|
||||
) {
|
||||
// CI waiting cannot revive an offline desktop; keep its proven migration
|
||||
// pending until that target control reconnects.
|
||||
return status
|
||||
}
|
||||
await deps.wait(config.pollIntervalMs)
|
||||
}
|
||||
throw new Error('timed out waiting for candidate migration')
|
||||
}
|
||||
|
||||
export async function setCellState(config, adminPost, cellId, enabled) {
|
||||
const post = async (path, body) => await adminPost(config.directorOrigin, path, body)
|
||||
const inspected = await inspectAdmissionSelector(post)
|
||||
if (inspected.selector.generation > 0) {
|
||||
await transitionAdmissionSelector(post, {
|
||||
[cellId]: enabled ? 'general' : 'existing-only'
|
||||
})
|
||||
return
|
||||
}
|
||||
await post('/v1/admin/cell-state', { v: 1, cellId, enabled })
|
||||
}
|
||||
|
||||
function assertNoDurableActivity(status, operation) {
|
||||
const activity = [
|
||||
status.assignments,
|
||||
status.activityLeases,
|
||||
status.reservedRequests,
|
||||
status.outgoingMigrations,
|
||||
status.incomingMigrations
|
||||
]
|
||||
if (activity.some((value) => Number(value) !== 0)) {
|
||||
throw new Error(`${operation} requires zero durable activity`)
|
||||
}
|
||||
}
|
||||
|
||||
async function recoverCandidateFailure(
|
||||
config,
|
||||
deps,
|
||||
adminPost,
|
||||
source,
|
||||
target,
|
||||
allowEmptyAdmissionRollback,
|
||||
selectorActive
|
||||
) {
|
||||
const status = await adminPost(config.directorOrigin, '/v1/admin/evacuation-status', {
|
||||
v: 1,
|
||||
sourceCellId: source.cellId,
|
||||
targetCellId: target.cellId,
|
||||
completeReady: false
|
||||
}).catch(() => null)
|
||||
if (!selectorActive && allowEmptyAdmissionRollback && status?.inProgress === 0) {
|
||||
await setCellState(config, adminPost, source.cellId, true).catch(() => undefined)
|
||||
await setCellState(config, adminPost, target.cellId, false).catch(() => undefined)
|
||||
return
|
||||
}
|
||||
if (!selectorActive && status && status.inProgress > 0 && status.targetRegistered === 0) {
|
||||
await setCellState(config, adminPost, source.cellId, true).catch(() => undefined)
|
||||
deps.emit({
|
||||
event: 'candidate_rollback_waiting_for_lease_expiry',
|
||||
sourceCellId: source.cellId,
|
||||
targetCellId: target.cellId,
|
||||
inProgress: status.inProgress
|
||||
})
|
||||
return
|
||||
}
|
||||
deps.emit({
|
||||
event: 'candidate_forward_recovery_required',
|
||||
sourceCellId: source.cellId,
|
||||
targetCellId: target.cellId,
|
||||
targetRegistered: status?.targetRegistered ?? null
|
||||
})
|
||||
}
|
||||
|
||||
export async function drainSource(
|
||||
config,
|
||||
deps,
|
||||
token,
|
||||
source,
|
||||
graceMs = config.drainGraceMs,
|
||||
traceValue
|
||||
) {
|
||||
const response = await deps.fetch(`${source.origin}/v1/admin/drain`, {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
authorization: `Bearer ${token}`,
|
||||
'content-type': 'application/json',
|
||||
...(traceValue ? { 'x-orca-drain-trace': traceValue } : {})
|
||||
},
|
||||
body: JSON.stringify({ v: 1, graceMs }),
|
||||
signal: AbortSignal.timeout(30_000)
|
||||
})
|
||||
await responseJson(response, 'source drain')
|
||||
return {
|
||||
backendStatus: response.status,
|
||||
backendInstance: response.headers.get('x-orca-backend-instance') ?? undefined
|
||||
}
|
||||
}
|
||||
|
||||
async function verifyCandidateCompletion(
|
||||
config,
|
||||
adminPost,
|
||||
source,
|
||||
target,
|
||||
event,
|
||||
eventName = 'candidate_complete'
|
||||
) {
|
||||
const finalSource = await adminPost(config.directorOrigin, '/v1/admin/cell-status', {
|
||||
v: 1,
|
||||
cellId: source.cellId
|
||||
})
|
||||
const finalTarget = await adminPost(config.directorOrigin, '/v1/admin/cell-status', {
|
||||
v: 1,
|
||||
cellId: target.cellId
|
||||
})
|
||||
if (
|
||||
finalSource.status.activityLeases !== 0 ||
|
||||
finalSource.status.reservedRequests !== 0 ||
|
||||
finalSource.status.outgoingMigrations !== 0 ||
|
||||
finalSource.status.runtime?.observedRequests !== 0 ||
|
||||
finalTarget.status.incomingMigrations !== 0 ||
|
||||
finalTarget.status.reservedRequests !== finalTarget.status.activityRequestUnits
|
||||
) {
|
||||
throw new Error('aggregate post-migration counts are not reconciled')
|
||||
}
|
||||
event({
|
||||
event: eventName,
|
||||
sourceCellId: source.cellId,
|
||||
targetCellId: target.cellId,
|
||||
dormantSourceAssignments: finalSource.status.assignments,
|
||||
targetAssignments: finalTarget.status.assignments,
|
||||
targetActivityLeases: finalTarget.status.activityLeases,
|
||||
targetReservedRequests: finalTarget.status.reservedRequests
|
||||
})
|
||||
}
|
||||
|
||||
export async function runCandidateDeployment(config, overrides = {}) {
|
||||
const deps = {
|
||||
commandJson: overrides.commandJson ?? defaultCommandJson,
|
||||
identityToken: overrides.identityToken ?? defaultIdentityToken,
|
||||
fetch: overrides.fetch ?? fetch,
|
||||
emit:
|
||||
overrides.emit ??
|
||||
((event) => process.stdout.write(`${JSON.stringify(event)}\n`)),
|
||||
now: overrides.now ?? Date.now,
|
||||
wait: overrides.wait ?? ((ms) => new Promise((resolve) => setTimeout(resolve, ms))),
|
||||
random: overrides.random ?? Math.random
|
||||
}
|
||||
const topology = JSON.parse(readFileSync(config.topologyFile, 'utf8'))
|
||||
const { source, target } = selectDeployments(
|
||||
topology,
|
||||
config.sourceCellId,
|
||||
config.targetCellId
|
||||
)
|
||||
const token = deps.identityToken(config.adminAudience)
|
||||
const adminPost = createAdminPost(config, deps, token)
|
||||
const selectorPost = async (path, body) =>
|
||||
await adminPost(config.directorOrigin, path, body)
|
||||
const selectorInspection = await inspectAdmissionSelector(selectorPost)
|
||||
const selectorActive = selectorInspection.selector.generation > 0
|
||||
const sourceStatus = await inspectCell(config, deps, adminPost, source)
|
||||
const targetStatus = await inspectCell(config, deps, adminPost, target)
|
||||
const sourceAdmission = selectorActive
|
||||
? selectorCellState(selectorInspection.selector, source.cellId)
|
||||
: sourceStatus.enabled
|
||||
? 'general'
|
||||
: 'existing-only'
|
||||
const targetAdmission = selectorActive
|
||||
? selectorCellState(selectorInspection.selector, target.cellId)
|
||||
: targetStatus.enabled
|
||||
? 'general'
|
||||
: 'existing-only'
|
||||
if (config.mode === 'audit') {
|
||||
const migration = await adminPost(config.directorOrigin, '/v1/admin/evacuation-status', {
|
||||
v: 1,
|
||||
sourceCellId: source.cellId,
|
||||
targetCellId: target.cellId,
|
||||
completeReady: false
|
||||
})
|
||||
// Forward recovery needs durable aggregate evidence without exposing assignment identities.
|
||||
deps.emit({
|
||||
event: 'candidate_audit',
|
||||
source: aggregateCellStatus(sourceStatus),
|
||||
target: aggregateCellStatus(targetStatus),
|
||||
migration
|
||||
})
|
||||
return
|
||||
}
|
||||
if (config.mode === 'recover-forward') {
|
||||
if (
|
||||
selectorActive
|
||||
? sourceAdmission !== 'existing-only' || targetAdmission !== 'migration-only'
|
||||
: sourceStatus.enabled || !targetStatus.enabled
|
||||
) {
|
||||
throw new Error(
|
||||
selectorActive
|
||||
? 'forward recovery requires existing-only source and migration-only target'
|
||||
: 'forward recovery requires disabled source and enabled target'
|
||||
)
|
||||
}
|
||||
await drainSource(config, deps, token, source)
|
||||
await waitForMigration(config, deps, adminPost, false)
|
||||
const completion = await waitForMigration(config, deps, adminPost, true)
|
||||
if (completion.inProgress > 0) {
|
||||
deps.emit({
|
||||
event: 'candidate_forward_pending',
|
||||
sourceCellId: source.cellId,
|
||||
targetCellId: target.cellId,
|
||||
inProgress: completion.inProgress,
|
||||
registeredSourceActive: completion.registeredSourceActive,
|
||||
registeredCompletable: completion.registeredCompletable,
|
||||
registeredTargetInactive: completion.registeredTargetInactive
|
||||
})
|
||||
throw new Error('forward recovery remains pending for inactive target controls')
|
||||
}
|
||||
await verifyCandidateCompletion(
|
||||
config,
|
||||
adminPost,
|
||||
source,
|
||||
target,
|
||||
deps.emit,
|
||||
'candidate_forward_recovered'
|
||||
)
|
||||
return
|
||||
}
|
||||
if (config.mode === 'disable-cell') {
|
||||
if (targetStatus.enabled) await setCellState(config, adminPost, target.cellId, false)
|
||||
// Disabling new admission preserves origin-owned sessions and durable recovery work.
|
||||
deps.emit({
|
||||
event: 'cell_admission_disabled',
|
||||
targetCellId: target.cellId,
|
||||
changed: targetStatus.enabled,
|
||||
assignments: targetStatus.assignments,
|
||||
activityLeases: targetStatus.activityLeases,
|
||||
reservedRequests: targetStatus.reservedRequests,
|
||||
outgoingMigrations: targetStatus.outgoingMigrations,
|
||||
incomingMigrations: targetStatus.incomingMigrations
|
||||
})
|
||||
return
|
||||
}
|
||||
// Repair is safe only before a candidate owns assignments or origin-scoped work.
|
||||
if (config.mode === 'reset-empty-candidate') {
|
||||
assertNoDurableActivity(targetStatus, 'candidate admission reset')
|
||||
if (targetStatus.enabled) await setCellState(config, adminPost, target.cellId, false)
|
||||
deps.emit({
|
||||
event: 'candidate_admission_reset',
|
||||
targetCellId: target.cellId,
|
||||
changed: targetStatus.enabled
|
||||
})
|
||||
return
|
||||
}
|
||||
if (config.mode === 'enable-empty-cell') {
|
||||
assertNoDurableActivity(targetStatus, 'cell admission enable')
|
||||
if (selectorActive ? targetAdmission === 'general' : targetStatus.enabled) {
|
||||
deps.emit({ event: 'cell_admission_enabled', targetCellId: target.cellId, changed: false })
|
||||
return
|
||||
}
|
||||
}
|
||||
const continuingEvacuation = config.mode === 'continue-evacuation'
|
||||
if (continuingEvacuation) {
|
||||
if (
|
||||
selectorActive
|
||||
? targetAdmission !== 'migration-only'
|
||||
: !targetStatus.enabled
|
||||
) {
|
||||
throw new Error(
|
||||
selectorActive
|
||||
? 'continued evacuation requires migration-only target'
|
||||
: 'continued evacuation requires enabled target'
|
||||
)
|
||||
}
|
||||
} else if (!selectorActive && targetStatus.enabled) {
|
||||
throw new Error('candidate cell is already enabled')
|
||||
} else if (
|
||||
selectorActive &&
|
||||
!['migration-only', 'existing-only'].includes(targetAdmission)
|
||||
) {
|
||||
throw new Error('candidate cell must not be generally admitted')
|
||||
}
|
||||
const capacity = await adminPost(config.directorOrigin, '/v1/admin/evacuation-capacity', {
|
||||
v: 1,
|
||||
sourceCellId: source.cellId,
|
||||
targetCellId: target.cellId
|
||||
})
|
||||
if (capacity.requiredTargetUnits > capacity.availableTargetUnits) {
|
||||
throw new Error('candidate lacks survivor request-unit headroom')
|
||||
}
|
||||
deps.emit({
|
||||
event: 'candidate_preflight',
|
||||
mode: config.mode,
|
||||
sourceCellId: source.cellId,
|
||||
targetCellId: target.cellId,
|
||||
sourceOrigin: source.origin,
|
||||
targetOrigin: target.origin,
|
||||
sourceMig: source.migName,
|
||||
targetMig: target.migName,
|
||||
sourceBackend: source.backendName,
|
||||
targetBackend: target.backendName,
|
||||
sourceDigest: source.imageDigest,
|
||||
targetDigest: target.imageDigest,
|
||||
sourceAssignments: capacity.sourceAssignments,
|
||||
requiredTargetUnits: capacity.requiredTargetUnits,
|
||||
availableTargetUnits: capacity.availableTargetUnits
|
||||
})
|
||||
if (config.mode === 'preflight') return
|
||||
if (config.mode === 'enable-empty-cell') {
|
||||
await setCellState(config, adminPost, target.cellId, true)
|
||||
deps.emit({ event: 'cell_admission_enabled', targetCellId: target.cellId, changed: true })
|
||||
return
|
||||
}
|
||||
// Fresh execution starts from source-only admission; continuation preserves its target.
|
||||
if (
|
||||
!continuingEvacuation &&
|
||||
(selectorActive ? sourceAdmission !== 'existing-only' : !sourceStatus.enabled)
|
||||
) {
|
||||
throw new Error(
|
||||
selectorActive ? 'source cell is not existing-only' : 'source cell is not enabled'
|
||||
)
|
||||
}
|
||||
if (selectorActive && targetAdmission !== 'migration-only') {
|
||||
throw new Error('target cell is not migration-only')
|
||||
}
|
||||
|
||||
let migrationsStarted = 0
|
||||
try {
|
||||
if (!selectorActive) {
|
||||
if (sourceStatus.enabled) await setCellState(config, adminPost, source.cellId, false)
|
||||
if (!targetStatus.enabled) await setCellState(config, adminPost, target.cellId, true)
|
||||
}
|
||||
for (;;) {
|
||||
const result = await adminPost(config.directorOrigin, '/v1/admin/evacuate-cell', {
|
||||
v: 1,
|
||||
sourceCellId: source.cellId,
|
||||
targetCellId: target.cellId,
|
||||
limit: config.batchSize
|
||||
})
|
||||
migrationsStarted += result.started
|
||||
deps.emit({ event: 'migration_batch', started: result.started, totalStarted: migrationsStarted })
|
||||
if (result.started === 0) break
|
||||
}
|
||||
} catch (error) {
|
||||
await recoverCandidateFailure(
|
||||
config,
|
||||
deps,
|
||||
adminPost,
|
||||
source,
|
||||
target,
|
||||
!continuingEvacuation,
|
||||
selectorActive
|
||||
)
|
||||
throw error
|
||||
}
|
||||
|
||||
try {
|
||||
if (selectorActive) {
|
||||
const currentSelector = await inspectAdmissionSelector(selectorPost)
|
||||
if (
|
||||
currentSelector.selector.generation !== selectorInspection.selector.generation ||
|
||||
JSON.stringify(currentSelector.selector.membership) !==
|
||||
JSON.stringify(selectorInspection.selector.membership)
|
||||
) {
|
||||
throw new Error('admission selector changed before drain')
|
||||
}
|
||||
}
|
||||
await drainSource(config, deps, token, source)
|
||||
await waitForMigration(config, deps, adminPost, false)
|
||||
const completion = await waitForMigration(config, deps, adminPost, true)
|
||||
if (completion.inProgress > 0) {
|
||||
throw new Error('candidate migration remains pending for inactive target controls')
|
||||
}
|
||||
} catch (error) {
|
||||
// A completion response can be lost after its transaction commits. Never
|
||||
// reverse admission here merely because no in-progress row remains.
|
||||
await recoverCandidateFailure(
|
||||
config,
|
||||
deps,
|
||||
adminPost,
|
||||
source,
|
||||
target,
|
||||
false,
|
||||
selectorActive
|
||||
)
|
||||
throw error
|
||||
}
|
||||
|
||||
await verifyCandidateCompletion(config, adminPost, source, target, deps.emit)
|
||||
}
|
||||
|
||||
export function aggregateCellStatus(status) {
|
||||
return {
|
||||
cellId: status.cellId,
|
||||
enabled: status.enabled,
|
||||
assignments: status.assignments,
|
||||
activityLeases: status.activityLeases,
|
||||
activityRequestUnits: status.activityRequestUnits,
|
||||
reservedRequests: status.reservedRequests,
|
||||
outgoingMigrations: status.outgoingMigrations,
|
||||
incomingMigrations: status.incomingMigrations,
|
||||
runtimeReady: status.runtime?.ready ?? false,
|
||||
heartbeatFresh: status.runtime?.heartbeatFresh ?? false,
|
||||
observedRequests: status.runtime?.observedRequests ?? null
|
||||
}
|
||||
}
|
||||
|
||||
export async function main(argv = process.argv.slice(2)) {
|
||||
await runCandidateDeployment(parseArguments(argv))
|
||||
}
|
||||
|
||||
if (import.meta.url === pathToFileURL(process.argv[1]).href) {
|
||||
main().catch((error) => {
|
||||
process.stderr.write(`${error instanceof Error ? error.message : String(error)}\n`)
|
||||
process.exitCode = 1
|
||||
})
|
||||
}
|
||||
@@ -1,922 +0,0 @@
|
||||
import assert from 'node:assert/strict'
|
||||
import { mkdtempSync, rmSync, writeFileSync } from 'node:fs'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { join } from 'node:path'
|
||||
import { test } from 'node:test'
|
||||
import {
|
||||
assertDeploymentConnectionCapacity,
|
||||
createAdminPost,
|
||||
deployment,
|
||||
parseArguments,
|
||||
runCandidateDeployment,
|
||||
selectDeployments,
|
||||
suppliedAdminIdentityToken,
|
||||
validateBackend,
|
||||
validateInstance,
|
||||
validateMig
|
||||
} from './deploy-relay-gce-candidate.mjs'
|
||||
|
||||
const runtimeServiceAccount = 'orca-relay@example.iam.gserviceaccount.com'
|
||||
const digestA = `sha256:${'a'.repeat(64)}`
|
||||
const digestB = `sha256:${'b'.repeat(64)}`
|
||||
|
||||
test('retries evacuation-status HTTP 500 without widening other admin retries', async () => {
|
||||
const events = []
|
||||
let calls = 0
|
||||
const adminPost = createAdminPost({}, {
|
||||
fetch: async () => {
|
||||
calls++
|
||||
return calls === 1
|
||||
? new Response(JSON.stringify({ error: 'transient' }), { status: 500 })
|
||||
: new Response(JSON.stringify({ ok: true }), { status: 200 })
|
||||
},
|
||||
emit: (event) => events.push(event),
|
||||
wait: async () => {},
|
||||
random: () => 0
|
||||
}, 'token')
|
||||
assert.deepEqual(
|
||||
await adminPost('https://relay.example', '/v1/admin/evacuation-status', {}),
|
||||
{ ok: true }
|
||||
)
|
||||
assert.equal(calls, 2)
|
||||
assert.deepEqual(events, [{
|
||||
event: 'candidate_admin_retry',
|
||||
path: '/v1/admin/evacuation-status',
|
||||
attempt: 1,
|
||||
reason: 'http_500'
|
||||
}])
|
||||
|
||||
calls = 0
|
||||
await assert.rejects(
|
||||
createAdminPost({}, {
|
||||
fetch: async () => {
|
||||
calls++
|
||||
return new Response(JSON.stringify({ error: 'persistent' }), { status: 500 })
|
||||
},
|
||||
emit: () => {},
|
||||
wait: async () => {},
|
||||
random: () => 0
|
||||
}, 'token')('https://relay.example', '/v1/admin/runtime-status', {}),
|
||||
/runtime-status failed: persistent/
|
||||
)
|
||||
assert.equal(calls, 1)
|
||||
})
|
||||
|
||||
test('a reserve-cell refusal names the flip back to db; other 404s keep their error', async () => {
|
||||
const post = (body) => createAdminPost({}, {
|
||||
fetch: async () => Response.json(body, { status: 404 }),
|
||||
emit: () => {},
|
||||
wait: async () => {},
|
||||
random: () => 0
|
||||
}, 'token')
|
||||
await assert.rejects(
|
||||
post({ error: 'cell_in_reserve_mode' })(
|
||||
'https://relay.example',
|
||||
'/v1/admin/cell-status',
|
||||
{ v: 1, cellId: 'production-gce-c7' }
|
||||
),
|
||||
/production-gce-c7 is in reserve mode .*flip it to admitMode=db/
|
||||
)
|
||||
await assert.rejects(
|
||||
post({ error: 'cell_not_found' })(
|
||||
'https://relay.example',
|
||||
'/v1/admin/cell-status',
|
||||
{ v: 1, cellId: 'production-gce-c7' }
|
||||
),
|
||||
/cell-status failed: cell_not_found/
|
||||
)
|
||||
await assert.rejects(
|
||||
post({ error: 'cell_in_reserve_mode' })(
|
||||
'https://relay.example',
|
||||
'/v1/admin/evacuation-status',
|
||||
{ v: 1, sourceCellId: 'production-gce-c7', targetCellId: 'production-gce-c8' }
|
||||
),
|
||||
/production-gce-c7 or production-gce-c8 is in reserve mode/
|
||||
)
|
||||
})
|
||||
|
||||
test('verifies a 1,000-cap cell against both runtime and director telemetry', () => {
|
||||
const expected = deployment(
|
||||
{
|
||||
...topology().target,
|
||||
connection_hard_cap: 1_000,
|
||||
connection_unobserved_bound: 60
|
||||
},
|
||||
'target'
|
||||
)
|
||||
const capacity = {
|
||||
hardCap: 1_000,
|
||||
controlRebindReserve: 100,
|
||||
ordinaryConnectionLimit: 900,
|
||||
unobservedBound: 60,
|
||||
normalAdmissionPause: 840
|
||||
}
|
||||
assert.doesNotThrow(() =>
|
||||
assertDeploymentConnectionCapacity(expected, capacity, {
|
||||
...capacity,
|
||||
heartbeatFresh: true
|
||||
})
|
||||
)
|
||||
assert.throws(
|
||||
() =>
|
||||
assertDeploymentConnectionCapacity(expected, capacity, {
|
||||
...capacity,
|
||||
hardCap: 600,
|
||||
heartbeatFresh: true
|
||||
}),
|
||||
/differs from Terraform/
|
||||
)
|
||||
})
|
||||
|
||||
test('verifies a 3,000-cap regional cell with a 2,840 placement boundary', () => {
|
||||
const expected = deployment(
|
||||
{
|
||||
...topology().target,
|
||||
connection_hard_cap: 3_000,
|
||||
connection_unobserved_bound: 60
|
||||
},
|
||||
'target'
|
||||
)
|
||||
const capacity = {
|
||||
hardCap: 3_000,
|
||||
controlRebindReserve: 100,
|
||||
ordinaryConnectionLimit: 2_900,
|
||||
unobservedBound: 60,
|
||||
normalAdmissionPause: 2_840
|
||||
}
|
||||
|
||||
assert.doesNotThrow(() =>
|
||||
assertDeploymentConnectionCapacity(expected, capacity, {
|
||||
...capacity,
|
||||
heartbeatFresh: true
|
||||
})
|
||||
)
|
||||
})
|
||||
|
||||
function topology() {
|
||||
return {
|
||||
source: {
|
||||
origin: 'https://c1.relay.example.com',
|
||||
zone: 'us-central1-b',
|
||||
mig_name: 'relay-c1',
|
||||
instance_group: 'https://compute.example/instanceGroups/relay-c1',
|
||||
backend_name: 'relay-c1',
|
||||
backend_id: 'https://compute.example/backendServices/relay-c1',
|
||||
image: `us-central1-docker.pkg.dev/project/repo/relay@${digestA}`,
|
||||
capacity_requests: 4_000,
|
||||
initially_enabled: true
|
||||
},
|
||||
target: {
|
||||
origin: 'https://c2.relay.example.com',
|
||||
zone: 'us-central1-c',
|
||||
mig_name: 'relay-c2',
|
||||
instance_group: 'https://compute.example/instanceGroups/relay-c2',
|
||||
backend_name: 'relay-c2',
|
||||
backend_id: 'https://compute.example/backendServices/relay-c2',
|
||||
image: `us-central1-docker.pkg.dev/project/repo/relay@${digestB}`,
|
||||
capacity_requests: 4_000,
|
||||
initially_enabled: false
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function withTopology(operation) {
|
||||
const directory = mkdtempSync(join(tmpdir(), 'relay-gce-candidate-'))
|
||||
const file = join(directory, 'topology.json')
|
||||
writeFileSync(file, JSON.stringify(topology()))
|
||||
return Promise.resolve(operation(file)).finally(() => rmSync(directory, { recursive: true }))
|
||||
}
|
||||
|
||||
function config(topologyFile, mode = 'preflight') {
|
||||
return {
|
||||
project: 'test-project',
|
||||
directorOrigin: 'https://relay.example.com',
|
||||
adminAudience: 'https://relay.example.com/v1/admin/drain',
|
||||
topologyFile,
|
||||
sourceCellId: 'source',
|
||||
targetCellId: 'target',
|
||||
runtimeServiceAccount,
|
||||
mode,
|
||||
batchSize: 100,
|
||||
drainGraceMs: 120_000,
|
||||
pollIntervalMs: 1,
|
||||
timeoutMs: 1_000
|
||||
}
|
||||
}
|
||||
|
||||
function response(body, status = 200) {
|
||||
return new Response(JSON.stringify(body), {
|
||||
status,
|
||||
headers: { 'content-type': 'application/json' }
|
||||
})
|
||||
}
|
||||
|
||||
function migrationResponse(body) {
|
||||
return response({
|
||||
registeredSourceActive: 0,
|
||||
registeredCompletable: 0,
|
||||
registeredTargetInactive: 0,
|
||||
expiredUnregistered: 0,
|
||||
repairableExpiredUnregistered: 0,
|
||||
abortableExpiredUnregistered: 0,
|
||||
blockedExpiredUnregistered: 0,
|
||||
blockedExpiredOnNewerTargetAssignment: 0,
|
||||
...body
|
||||
})
|
||||
}
|
||||
|
||||
function fakeCommand(args) {
|
||||
const name = args[args.indexOf('describe') + 1] ?? args[args.indexOf('list-instances') + 1]
|
||||
if (args.includes('list-instances')) {
|
||||
return [
|
||||
{
|
||||
instance: `https://compute.example/instances/${name}-vm`,
|
||||
instanceStatus: 'RUNNING',
|
||||
currentAction: 'NONE'
|
||||
}
|
||||
]
|
||||
}
|
||||
if (args.includes('instance-groups')) {
|
||||
return {
|
||||
targetSize: 1,
|
||||
updatePolicy: {
|
||||
replacementMethod: 'RECREATE',
|
||||
maxSurge: { fixed: 0 },
|
||||
maxUnavailable: { fixed: 1 }
|
||||
}
|
||||
}
|
||||
}
|
||||
if (args.includes('instances')) {
|
||||
return {
|
||||
networkInterfaces: [{ networkIP: '10.42.0.2' }],
|
||||
serviceAccounts: [{ email: runtimeServiceAccount }]
|
||||
}
|
||||
}
|
||||
const cell = name === 'relay-c1' ? topology().source : topology().target
|
||||
return { protocol: 'HTTP', timeoutSec: 86_400, backends: [{ group: cell.instance_group }] }
|
||||
}
|
||||
|
||||
function harness({
|
||||
failTargetEnable = false,
|
||||
failDrain = false,
|
||||
failAfterRegistration = false,
|
||||
failCompletionResponse = false,
|
||||
sourceEnabled = true,
|
||||
targetEnabled = false,
|
||||
targetAssignments = 0,
|
||||
migrationInProgress = 0,
|
||||
migrationTargetRegistered = 0,
|
||||
migrationTargetInactive = 0,
|
||||
transientCompletionFailures = 0,
|
||||
dormantSourceAssignments = 0,
|
||||
selectorGeneration = 0
|
||||
} = {}) {
|
||||
const state = {
|
||||
source: {
|
||||
enabled: selectorGeneration > 0 ? false : sourceEnabled,
|
||||
assignments: 2,
|
||||
activityLeases: 2
|
||||
},
|
||||
target: {
|
||||
enabled: selectorGeneration > 0 ? true : targetEnabled,
|
||||
assignments: targetAssignments,
|
||||
activityLeases: targetAssignments
|
||||
}
|
||||
}
|
||||
const events = []
|
||||
const stateChanges = []
|
||||
let batch = 0
|
||||
let migrationCompleted = false
|
||||
let remainingTransientCompletionFailures = transientCompletionFailures
|
||||
const fetch = async (url, options = {}) => {
|
||||
const parsed = new URL(url)
|
||||
if (parsed.pathname === '/health' || parsed.pathname === '/ready') return response({ ok: true })
|
||||
const body = JSON.parse(options.body ?? '{}')
|
||||
if (parsed.pathname === '/v1/admin/runtime-status') {
|
||||
const target = parsed.origin.includes('c2.')
|
||||
return response({
|
||||
v: 1,
|
||||
role: 'cell',
|
||||
cellId: target ? 'target' : 'source',
|
||||
cellUrl: parsed.origin,
|
||||
imageDigest: target ? digestB : digestA
|
||||
})
|
||||
}
|
||||
if (parsed.pathname === '/v1/admin/admission-selector/status') {
|
||||
return response({
|
||||
v: 1,
|
||||
selector: {
|
||||
generation: selectorGeneration,
|
||||
attemptId: null,
|
||||
membership: {
|
||||
existingOnly:
|
||||
selectorGeneration > 0
|
||||
? ['source']
|
||||
: Object.keys(state).filter((cellId) => !state[cellId].enabled),
|
||||
migrationOnly: selectorGeneration > 0 ? ['target'] : [],
|
||||
general:
|
||||
selectorGeneration > 0
|
||||
? []
|
||||
: Object.keys(state).filter((cellId) => state[cellId].enabled)
|
||||
}
|
||||
},
|
||||
intent: null
|
||||
})
|
||||
}
|
||||
if (parsed.pathname === '/v1/admin/cell-status') {
|
||||
const cell = state[body.cellId]
|
||||
return response({
|
||||
v: 1,
|
||||
status: {
|
||||
cellId: body.cellId,
|
||||
cellUrl: topology()[body.cellId].origin,
|
||||
enabled: cell.enabled,
|
||||
admissionState:
|
||||
selectorGeneration > 0
|
||||
? body.cellId === 'source'
|
||||
? 'existing-only'
|
||||
: 'migration-only'
|
||||
: cell.enabled
|
||||
? 'general'
|
||||
: 'existing-only',
|
||||
assignments: cell.assignments,
|
||||
activityLeases: cell.activityLeases,
|
||||
activityRequestUnits: cell.activityLeases,
|
||||
reservedRequests: cell.activityLeases,
|
||||
outgoingMigrations: 0,
|
||||
incomingMigrations: 0,
|
||||
runtime: {
|
||||
cellUrl: topology()[body.cellId].origin,
|
||||
ready: true,
|
||||
heartbeatFresh: true,
|
||||
observedRequests: cell.activityLeases
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
if (parsed.pathname === '/v1/admin/evacuation-capacity') {
|
||||
return response({ sourceAssignments: 2, requiredTargetUnits: 4, availableTargetUnits: 4_000 })
|
||||
}
|
||||
if (parsed.pathname === '/v1/admin/cell-state') {
|
||||
stateChanges.push([body.cellId, body.enabled])
|
||||
if (failTargetEnable && body.cellId === 'target' && body.enabled) {
|
||||
return response({ error: 'injected_enable_failure' }, 409)
|
||||
}
|
||||
state[body.cellId].enabled = body.enabled
|
||||
return response({ ok: true })
|
||||
}
|
||||
if (parsed.pathname === '/v1/admin/evacuate-cell') {
|
||||
if (failAfterRegistration && batch > 0) {
|
||||
return response({ error: 'injected_batch_failure' }, 503)
|
||||
}
|
||||
const started = batch++ === 0 ? 2 : 0
|
||||
return response({ v: 1, started })
|
||||
}
|
||||
if (parsed.pathname === '/v1/admin/evacuation-status') {
|
||||
if (failAfterRegistration) {
|
||||
return migrationResponse({
|
||||
v: 1,
|
||||
inProgress: 2,
|
||||
targetRegistered: 1,
|
||||
registeredSourceActive: 1,
|
||||
completed: 0,
|
||||
blocked: 1
|
||||
})
|
||||
}
|
||||
if (failDrain) {
|
||||
return migrationResponse({
|
||||
v: 1,
|
||||
inProgress: 2,
|
||||
targetRegistered: 0,
|
||||
completed: 0,
|
||||
blocked: 0
|
||||
})
|
||||
}
|
||||
if (body.completeReady) {
|
||||
state.source.assignments = dormantSourceAssignments
|
||||
state.source.activityLeases = 0
|
||||
state.target.assignments = 2
|
||||
state.target.activityLeases = 2
|
||||
if (remainingTransientCompletionFailures > 0) {
|
||||
remainingTransientCompletionFailures--
|
||||
throw new TypeError('injected transient fetch failure')
|
||||
}
|
||||
if (migrationTargetInactive > 0) {
|
||||
return migrationResponse({
|
||||
v: 1,
|
||||
inProgress: migrationTargetInactive,
|
||||
targetRegistered: migrationTargetInactive,
|
||||
registeredTargetInactive: migrationTargetInactive,
|
||||
completed: 0,
|
||||
blocked: migrationTargetInactive
|
||||
})
|
||||
}
|
||||
migrationCompleted = true
|
||||
if (failCompletionResponse) {
|
||||
return response({ error: 'injected_completion_response_failure' }, 503)
|
||||
}
|
||||
return migrationResponse({
|
||||
v: 1,
|
||||
inProgress: 0,
|
||||
targetRegistered: 0,
|
||||
completed: 2,
|
||||
blocked: 0
|
||||
})
|
||||
}
|
||||
if (migrationCompleted) {
|
||||
return migrationResponse({
|
||||
v: 1,
|
||||
inProgress: 0,
|
||||
targetRegistered: 0,
|
||||
completed: 0,
|
||||
blocked: 0
|
||||
})
|
||||
}
|
||||
if (batch === 0) {
|
||||
return migrationResponse({
|
||||
v: 1,
|
||||
inProgress: migrationInProgress,
|
||||
targetRegistered: migrationTargetRegistered,
|
||||
completed: 0,
|
||||
blocked: 0
|
||||
})
|
||||
}
|
||||
return migrationResponse({
|
||||
v: 1,
|
||||
inProgress: 2,
|
||||
targetRegistered: 2,
|
||||
registeredCompletable: 2,
|
||||
completed: 0,
|
||||
blocked: 0
|
||||
})
|
||||
}
|
||||
if (parsed.pathname === '/v1/admin/drain') {
|
||||
return failDrain ? response({ error: 'injected_drain_failure' }, 503) : response({ ok: true })
|
||||
}
|
||||
return response({ error: 'unexpected_request' }, 500)
|
||||
}
|
||||
return {
|
||||
overrides: {
|
||||
commandJson: fakeCommand,
|
||||
identityToken: () => 'secret-token-never-emitted',
|
||||
fetch,
|
||||
emit: (event) => events.push(event),
|
||||
wait: async () => undefined,
|
||||
random: () => 0
|
||||
},
|
||||
events,
|
||||
stateChanges,
|
||||
state
|
||||
}
|
||||
}
|
||||
|
||||
test('requires explicit dry-run/execute inputs and a distinct disabled candidate', () => {
|
||||
assert.throws(() => parseArguments([]), /missing --project/)
|
||||
assert.throws(
|
||||
() =>
|
||||
parseArguments([
|
||||
'--project',
|
||||
'project',
|
||||
'--director-origin',
|
||||
'https://relay.example.com',
|
||||
'--admin-audience',
|
||||
'https://relay.example.com/not-drain',
|
||||
'--topology-file',
|
||||
'topology.json',
|
||||
'--source-cell-id',
|
||||
'source',
|
||||
'--target-cell-id',
|
||||
'target',
|
||||
'--runtime-service-account',
|
||||
runtimeServiceAccount,
|
||||
'--mode',
|
||||
'preflight'
|
||||
]),
|
||||
/director drain URL/
|
||||
)
|
||||
assert.throws(() => selectDeployments(topology(), 'source', 'source'), /must differ/)
|
||||
const overlapping = topology()
|
||||
overlapping.target.backend_id = overlapping.source.backend_id
|
||||
assert.throws(() => selectDeployments(overlapping, 'source', 'target'), /backendId overlap/)
|
||||
const enabled = topology()
|
||||
enabled.target.initially_enabled = true
|
||||
assert.throws(() => selectDeployments(enabled, 'source', 'target'), /initially disabled/)
|
||||
})
|
||||
|
||||
test('accepts only a bounded JWT-shaped supplied admin identity token', () => {
|
||||
assert.equal(suppliedAdminIdentityToken({}), null)
|
||||
assert.equal(suppliedAdminIdentityToken({ ORCA_RELAY_ADMIN_ID_TOKEN: 'aaa.bbb.ccc' }), 'aaa.bbb.ccc')
|
||||
assert.throws(() => suppliedAdminIdentityToken({ ORCA_RELAY_ADMIN_ID_TOKEN: '' }))
|
||||
assert.throws(() => suppliedAdminIdentityToken({ ORCA_RELAY_ADMIN_ID_TOKEN: 'not-a-jwt' }))
|
||||
assert.throws(() =>
|
||||
suppliedAdminIdentityToken({ ORCA_RELAY_ADMIN_ID_TOKEN: `aaa.${'b'.repeat(8_190)}.ccc` })
|
||||
)
|
||||
})
|
||||
|
||||
test('rejects unsafe fixed-one topology, public IPs, and backend overlap', () => {
|
||||
const expected = selectDeployments(topology(), 'source', 'target').target
|
||||
assert.throws(() => validateMig({ targetSize: 2 }, [], expected), /fixed-one/)
|
||||
assert.throws(
|
||||
() =>
|
||||
validateInstance(
|
||||
{
|
||||
networkInterfaces: [{ accessConfigs: [{ natIP: '203.0.113.1' }] }],
|
||||
serviceAccounts: [{ email: runtimeServiceAccount }]
|
||||
},
|
||||
expected,
|
||||
runtimeServiceAccount
|
||||
),
|
||||
/public IP/
|
||||
)
|
||||
assert.throws(
|
||||
() => validateBackend({ protocol: 'HTTP', timeoutSec: 86_400, backends: [] }, expected),
|
||||
/topology mismatch/
|
||||
)
|
||||
})
|
||||
|
||||
test('preflights exact served digests and survivor headroom without mutating admission', async () => {
|
||||
await withTopology(async (file) => {
|
||||
const { overrides, events, stateChanges } = harness()
|
||||
await runCandidateDeployment(config(file), overrides)
|
||||
assert.deepEqual(stateChanges, [])
|
||||
assert.equal(events[0].event, 'candidate_preflight')
|
||||
assert.equal(events[0].targetDigest, digestB)
|
||||
assert.equal(JSON.stringify(events).includes('secret-token'), false)
|
||||
})
|
||||
})
|
||||
|
||||
test('audits a partially committed migration without changing admission or completing rows', async () => {
|
||||
await withTopology(async (file) => {
|
||||
const { overrides, events, stateChanges } = harness({
|
||||
sourceEnabled: false,
|
||||
targetEnabled: true,
|
||||
targetAssignments: 1,
|
||||
migrationInProgress: 2,
|
||||
migrationTargetRegistered: 2
|
||||
})
|
||||
await runCandidateDeployment(config(file, 'audit'), overrides)
|
||||
assert.deepEqual(stateChanges, [])
|
||||
assert.deepEqual(events, [
|
||||
{
|
||||
event: 'candidate_audit',
|
||||
source: {
|
||||
cellId: 'source',
|
||||
enabled: false,
|
||||
assignments: 2,
|
||||
activityLeases: 2,
|
||||
activityRequestUnits: 2,
|
||||
reservedRequests: 2,
|
||||
outgoingMigrations: 0,
|
||||
incomingMigrations: 0,
|
||||
runtimeReady: true,
|
||||
heartbeatFresh: true,
|
||||
observedRequests: 2
|
||||
},
|
||||
target: {
|
||||
cellId: 'target',
|
||||
enabled: true,
|
||||
assignments: 1,
|
||||
activityLeases: 1,
|
||||
activityRequestUnits: 1,
|
||||
reservedRequests: 1,
|
||||
outgoingMigrations: 0,
|
||||
incomingMigrations: 0,
|
||||
runtimeReady: true,
|
||||
heartbeatFresh: true,
|
||||
observedRequests: 1
|
||||
},
|
||||
migration: {
|
||||
v: 1,
|
||||
inProgress: 2,
|
||||
targetRegistered: 2,
|
||||
registeredSourceActive: 0,
|
||||
registeredCompletable: 0,
|
||||
registeredTargetInactive: 0,
|
||||
completed: 0,
|
||||
blocked: 0,
|
||||
expiredUnregistered: 0,
|
||||
repairableExpiredUnregistered: 0,
|
||||
abortableExpiredUnregistered: 0,
|
||||
blockedExpiredUnregistered: 0,
|
||||
blockedExpiredOnNewerTargetAssignment: 0
|
||||
}
|
||||
}
|
||||
])
|
||||
})
|
||||
})
|
||||
|
||||
test('preflights with source admission disabled but still refuses execution', async () => {
|
||||
await withTopology(async (file) => {
|
||||
const { overrides, events, stateChanges } = harness({ sourceEnabled: false })
|
||||
await runCandidateDeployment(config(file), overrides)
|
||||
assert.deepEqual(stateChanges, [])
|
||||
assert.equal(events.at(-1).event, 'candidate_preflight')
|
||||
await assert.rejects(
|
||||
runCandidateDeployment(config(file, 'execute'), overrides),
|
||||
/source cell is not enabled/
|
||||
)
|
||||
assert.deepEqual(stateChanges, [])
|
||||
})
|
||||
})
|
||||
|
||||
test('explicitly resets only an empty declared candidate to disabled admission', async () => {
|
||||
await withTopology(async (file) => {
|
||||
const { overrides, events, stateChanges } = harness({
|
||||
sourceEnabled: false,
|
||||
targetEnabled: true
|
||||
})
|
||||
await runCandidateDeployment(config(file, 'reset-empty-candidate'), overrides)
|
||||
assert.deepEqual(stateChanges, [['target', false]])
|
||||
assert.deepEqual(events.at(-1), {
|
||||
event: 'candidate_admission_reset',
|
||||
targetCellId: 'target',
|
||||
changed: true
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
test('refuses to reset candidate admission while it owns durable activity', async () => {
|
||||
await withTopology(async (file) => {
|
||||
const { overrides, stateChanges } = harness({ targetEnabled: true, targetAssignments: 1 })
|
||||
await assert.rejects(
|
||||
runCandidateDeployment(config(file, 'reset-empty-candidate'), overrides),
|
||||
/requires zero durable activity/
|
||||
)
|
||||
assert.deepEqual(stateChanges, [])
|
||||
})
|
||||
})
|
||||
|
||||
test('disables only new admission while preserving durable candidate activity', async () => {
|
||||
await withTopology(async (file) => {
|
||||
const { overrides, events, stateChanges } = harness({
|
||||
targetEnabled: true,
|
||||
targetAssignments: 1
|
||||
})
|
||||
await runCandidateDeployment(config(file, 'disable-cell'), overrides)
|
||||
assert.deepEqual(stateChanges, [['target', false]])
|
||||
assert.deepEqual(events.at(-1), {
|
||||
event: 'cell_admission_disabled',
|
||||
targetCellId: 'target',
|
||||
changed: true,
|
||||
assignments: 1,
|
||||
activityLeases: 1,
|
||||
reservedRequests: 1,
|
||||
outgoingMigrations: 0,
|
||||
incomingMigrations: 0
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
test('explicitly enables only an empty preflighted cell for admission', async () => {
|
||||
await withTopology(async (file) => {
|
||||
const { overrides, events, stateChanges } = harness({ sourceEnabled: false })
|
||||
await runCandidateDeployment(config(file, 'enable-empty-cell'), overrides)
|
||||
assert.deepEqual(stateChanges, [['target', true]])
|
||||
assert.equal(events.at(-2).event, 'candidate_preflight')
|
||||
assert.deepEqual(events.at(-1), {
|
||||
event: 'cell_admission_enabled',
|
||||
targetCellId: 'target',
|
||||
changed: true
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
test('refuses to enable cell admission while it owns durable activity', async () => {
|
||||
await withTopology(async (file) => {
|
||||
const { overrides, stateChanges } = harness({ targetAssignments: 1 })
|
||||
await assert.rejects(
|
||||
runCandidateDeployment(config(file, 'enable-empty-cell'), overrides),
|
||||
/requires zero durable activity/
|
||||
)
|
||||
assert.deepEqual(stateChanges, [])
|
||||
})
|
||||
})
|
||||
|
||||
test('executes target-first evacuation and verifies aggregate drained counts', async () => {
|
||||
await withTopology(async (file) => {
|
||||
const { overrides, events, stateChanges } = harness()
|
||||
await runCandidateDeployment(config(file, 'execute'), overrides)
|
||||
assert.deepEqual(stateChanges.slice(0, 2), [
|
||||
['source', false],
|
||||
['target', true]
|
||||
])
|
||||
assert.equal(events.at(-1).event, 'candidate_complete')
|
||||
assert.equal(events.at(-1).targetAssignments, 2)
|
||||
})
|
||||
})
|
||||
|
||||
test('executes within selector membership without legacy admission writes', async () => {
|
||||
await withTopology(async (file) => {
|
||||
const testHarness = harness({ selectorGeneration: 1 })
|
||||
await runCandidateDeployment(config(file, 'execute'), testHarness.overrides)
|
||||
assert.deepEqual(testHarness.stateChanges, [])
|
||||
assert.equal(testHarness.state.source.enabled, false)
|
||||
assert.equal(testHarness.state.target.enabled, true)
|
||||
})
|
||||
})
|
||||
|
||||
test('never restores legacy general admission after selector-era failure', async () => {
|
||||
await withTopology(async (file) => {
|
||||
const testHarness = harness({
|
||||
selectorGeneration: 1,
|
||||
failAfterRegistration: true
|
||||
})
|
||||
await assert.rejects(
|
||||
runCandidateDeployment(config(file, 'execute'), testHarness.overrides),
|
||||
/injected_batch_failure/
|
||||
)
|
||||
assert.deepEqual(testHarness.stateChanges, [])
|
||||
assert.equal(testHarness.state.source.enabled, false)
|
||||
})
|
||||
})
|
||||
|
||||
test('continues a partial evacuation without resetting target admission', async () => {
|
||||
await withTopology(async (file) => {
|
||||
const { overrides, events, stateChanges } = harness({
|
||||
sourceEnabled: true,
|
||||
targetEnabled: true,
|
||||
targetAssignments: 1,
|
||||
migrationInProgress: 1,
|
||||
migrationTargetRegistered: 1
|
||||
})
|
||||
await runCandidateDeployment(config(file, 'continue-evacuation'), overrides)
|
||||
assert.deepEqual(stateChanges, [['source', false]])
|
||||
assert.equal(events.at(-1).event, 'candidate_complete')
|
||||
})
|
||||
})
|
||||
|
||||
test('refuses continued evacuation unless target admission is already enabled', async () => {
|
||||
await withTopology(async (file) => {
|
||||
const { overrides, stateChanges } = harness()
|
||||
await assert.rejects(
|
||||
runCandidateDeployment(config(file, 'continue-evacuation'), overrides),
|
||||
/requires enabled target/
|
||||
)
|
||||
assert.deepEqual(stateChanges, [])
|
||||
})
|
||||
})
|
||||
|
||||
test('preserves partial target admission when continued batching fails', async () => {
|
||||
await withTopology(async (file) => {
|
||||
const { overrides, events, stateChanges } = harness({
|
||||
failAfterRegistration: true,
|
||||
sourceEnabled: true,
|
||||
targetEnabled: true,
|
||||
targetAssignments: 1,
|
||||
migrationInProgress: 1,
|
||||
migrationTargetRegistered: 1
|
||||
})
|
||||
await assert.rejects(
|
||||
runCandidateDeployment(config(file, 'continue-evacuation'), overrides),
|
||||
/injected_batch_failure/
|
||||
)
|
||||
assert.deepEqual(stateChanges, [['source', false]])
|
||||
assert.equal(events.at(-1).event, 'candidate_forward_recovery_required')
|
||||
})
|
||||
})
|
||||
|
||||
test('resumes only a committed forward migration and preserves dormant source assignments', async () => {
|
||||
await withTopology(async (file) => {
|
||||
const { overrides, events, stateChanges } = harness({
|
||||
sourceEnabled: false,
|
||||
targetEnabled: true,
|
||||
migrationInProgress: 2,
|
||||
migrationTargetRegistered: 2,
|
||||
dormantSourceAssignments: 7
|
||||
})
|
||||
await runCandidateDeployment(config(file, 'recover-forward'), overrides)
|
||||
assert.deepEqual(stateChanges, [])
|
||||
assert.deepEqual(events.at(-1), {
|
||||
event: 'candidate_forward_recovered',
|
||||
sourceCellId: 'source',
|
||||
targetCellId: 'target',
|
||||
dormantSourceAssignments: 7,
|
||||
targetAssignments: 2,
|
||||
targetActivityLeases: 2,
|
||||
targetReservedRequests: 2
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
test('retries a transient idempotent completion request without reversing admission', async () => {
|
||||
await withTopology(async (file) => {
|
||||
const { overrides, events, stateChanges } = harness({
|
||||
sourceEnabled: false,
|
||||
targetEnabled: true,
|
||||
migrationInProgress: 2,
|
||||
migrationTargetRegistered: 2,
|
||||
transientCompletionFailures: 1
|
||||
})
|
||||
await runCandidateDeployment(config(file, 'recover-forward'), overrides)
|
||||
assert.deepEqual(stateChanges, [])
|
||||
assert.deepEqual(
|
||||
events.filter(({ event }) => event === 'candidate_admin_retry'),
|
||||
[
|
||||
{
|
||||
event: 'candidate_admin_retry',
|
||||
path: '/v1/admin/evacuation-status',
|
||||
attempt: 1,
|
||||
reason: 'transport'
|
||||
}
|
||||
]
|
||||
)
|
||||
assert.equal(events.at(-1).event, 'candidate_forward_recovered')
|
||||
})
|
||||
})
|
||||
|
||||
test('stops forward recovery promptly when only registered offline targets remain', async () => {
|
||||
await withTopology(async (file) => {
|
||||
const { overrides, events, stateChanges } = harness({
|
||||
sourceEnabled: false,
|
||||
targetEnabled: true,
|
||||
migrationInProgress: 2,
|
||||
migrationTargetRegistered: 2,
|
||||
migrationTargetInactive: 2
|
||||
})
|
||||
await assert.rejects(
|
||||
runCandidateDeployment(config(file, 'recover-forward'), overrides),
|
||||
/pending for inactive target controls/
|
||||
)
|
||||
assert.deepEqual(stateChanges, [])
|
||||
assert.deepEqual(events.at(-1), {
|
||||
event: 'candidate_forward_pending',
|
||||
sourceCellId: 'source',
|
||||
targetCellId: 'target',
|
||||
inProgress: 2,
|
||||
registeredSourceActive: 0,
|
||||
registeredCompletable: 0,
|
||||
registeredTargetInactive: 2
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
test('refuses forward recovery unless source and target admission match committed direction', async () => {
|
||||
await withTopology(async (file) => {
|
||||
const { overrides, stateChanges } = harness()
|
||||
await assert.rejects(
|
||||
runCandidateDeployment(config(file, 'recover-forward'), overrides),
|
||||
/requires disabled source and enabled target/
|
||||
)
|
||||
assert.deepEqual(stateChanges, [])
|
||||
})
|
||||
})
|
||||
|
||||
test('re-enables an intact source when candidate admission fails before migration', async () => {
|
||||
await withTopology(async (file) => {
|
||||
const { overrides, stateChanges } = harness({ failTargetEnable: true })
|
||||
await assert.rejects(
|
||||
runCandidateDeployment(config(file, 'execute'), overrides),
|
||||
/injected_enable_failure/
|
||||
)
|
||||
assert.deepEqual(stateChanges, [
|
||||
['source', false],
|
||||
['target', true],
|
||||
['source', true],
|
||||
['target', false]
|
||||
])
|
||||
})
|
||||
})
|
||||
|
||||
test('re-enables the source and waits for lease rollback when no target registered', async () => {
|
||||
await withTopology(async (file) => {
|
||||
const { overrides, events, stateChanges } = harness({ failDrain: true })
|
||||
await assert.rejects(
|
||||
runCandidateDeployment(config(file, 'execute'), overrides),
|
||||
/injected_drain_failure/
|
||||
)
|
||||
assert.deepEqual(stateChanges.slice(-1), [['source', true]])
|
||||
assert.equal(events.at(-1).event, 'candidate_rollback_waiting_for_lease_expiry')
|
||||
})
|
||||
})
|
||||
|
||||
test('preserves both routes for forward recovery after a target registration', async () => {
|
||||
await withTopology(async (file) => {
|
||||
const { overrides, events, stateChanges } = harness({ failAfterRegistration: true })
|
||||
await assert.rejects(
|
||||
runCandidateDeployment(config(file, 'execute'), overrides),
|
||||
/injected_batch_failure/
|
||||
)
|
||||
assert.deepEqual(stateChanges, [
|
||||
['source', false],
|
||||
['target', true]
|
||||
])
|
||||
assert.equal(events.at(-1).event, 'candidate_forward_recovery_required')
|
||||
assert.equal(events.at(-1).targetRegistered, 1)
|
||||
})
|
||||
})
|
||||
|
||||
test('does not reverse admission after completion commits but its response is lost', async () => {
|
||||
await withTopology(async (file) => {
|
||||
const { overrides, events, stateChanges } = harness({ failCompletionResponse: true })
|
||||
await assert.rejects(
|
||||
runCandidateDeployment(config(file, 'execute'), overrides),
|
||||
/injected_completion_response_failure/
|
||||
)
|
||||
assert.deepEqual(stateChanges, [
|
||||
['source', false],
|
||||
['target', true]
|
||||
])
|
||||
assert.equal(events.at(-1).event, 'candidate_forward_recovery_required')
|
||||
assert.equal(events.at(-1).targetRegistered, 0)
|
||||
})
|
||||
})
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -1,136 +0,0 @@
|
||||
const REQUEST_TIMEOUT_MS = 15_000
|
||||
const JWT_PATTERN = /^[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+$/
|
||||
|
||||
export async function requestGitHubSmokeTokens(
|
||||
authOrigin,
|
||||
fetchImpl = fetch,
|
||||
environment = process.env,
|
||||
options = {}
|
||||
) {
|
||||
const origin = canonicalHttpsOrigin(authOrigin)
|
||||
const requestUrl = environment.ACTIONS_ID_TOKEN_REQUEST_URL
|
||||
const requestToken = environment.ACTIONS_ID_TOKEN_REQUEST_TOKEN
|
||||
if (!requestUrl || !requestToken) throw new Error('GitHub OIDC request context is unavailable')
|
||||
const audience = `${origin}/v1/internal/github-smoke-token`
|
||||
const oidcUrl = new URL(requestUrl)
|
||||
oidcUrl.searchParams.set('audience', audience)
|
||||
const oidcResponse = await fetchImpl(oidcUrl, {
|
||||
headers: { authorization: `Bearer ${requestToken}` },
|
||||
signal: AbortSignal.timeout(REQUEST_TIMEOUT_MS)
|
||||
})
|
||||
const oidc = await readJson(oidcResponse, 'GitHub OIDC request')
|
||||
if (!oidcResponse.ok || !validJwt(oidc.value)) {
|
||||
throw new Error(`GitHub OIDC request failed with ${oidcResponse.status}`)
|
||||
}
|
||||
const exchangeResponse = await fetchImpl(audience, {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
authorization: `Bearer ${oidc.value}`,
|
||||
...(options.relayAsiaLoad ? { 'content-type': 'application/json' } : {})
|
||||
},
|
||||
...(options.relayAsiaLoad
|
||||
? { body: JSON.stringify({ relayAsiaLoad: parseRelayAsiaLoadOptions(options.relayAsiaLoad) }) }
|
||||
: {}),
|
||||
signal: AbortSignal.timeout(REQUEST_TIMEOUT_MS)
|
||||
})
|
||||
const exchange = await readJson(exchangeResponse, 'Orca smoke identity exchange')
|
||||
if (!exchangeResponse.ok) {
|
||||
throw new Error(`Orca smoke identity exchange failed with ${exchangeResponse.status}`)
|
||||
}
|
||||
return {
|
||||
...parseAccessTokens(exchange.accessTokens),
|
||||
...(options.relayAsiaLoad
|
||||
? {
|
||||
relayAsiaLoadPrincipals: parseRelayAsiaLoadPrincipals(
|
||||
exchange.relayAsiaLoadPrincipals,
|
||||
options.relayAsiaLoad.principalCount
|
||||
)
|
||||
}
|
||||
: {})
|
||||
}
|
||||
}
|
||||
|
||||
// Staging auth mints Asia load principal sets for this many shards.
|
||||
export const RELAY_ASIA_LOAD_PRINCIPAL_SHARDS = 4
|
||||
|
||||
function parseRelayAsiaLoadOptions(value) {
|
||||
if (
|
||||
!value || typeof value !== 'object' ||
|
||||
!Number.isSafeInteger(value.shardIndex) || value.shardIndex < 0 || value.shardIndex >= RELAY_ASIA_LOAD_PRINCIPAL_SHARDS ||
|
||||
!Number.isSafeInteger(value.principalCount) || value.principalCount < 1 ||
|
||||
value.principalCount > 32
|
||||
) throw new Error('Relay Asia load principal request is invalid')
|
||||
return { v: 1, shardIndex: value.shardIndex, principalCount: value.principalCount }
|
||||
}
|
||||
|
||||
function canonicalHttpsOrigin(value) {
|
||||
const url = new URL(value)
|
||||
if (url.protocol !== 'https:' || url.pathname !== '/' || url.search || url.hash) {
|
||||
throw new Error('auth origin must be canonical HTTPS')
|
||||
}
|
||||
return url.origin
|
||||
}
|
||||
|
||||
function validJwt(value) {
|
||||
return typeof value === 'string' && value.length <= 8192 && JWT_PATTERN.test(value)
|
||||
}
|
||||
|
||||
function parseAccessTokens(value) {
|
||||
if (!value || typeof value !== 'object' || Array.isArray(value)) {
|
||||
throw new Error('Orca smoke identity response is malformed')
|
||||
}
|
||||
const expected = ['outsider', 'owner', 'recipient']
|
||||
if (Object.keys(value).sort().join(',') !== expected.join(',')) {
|
||||
throw new Error('Orca smoke identity response principals are invalid')
|
||||
}
|
||||
return Object.fromEntries(
|
||||
expected.map((name) => {
|
||||
const principal = value[name]
|
||||
if (
|
||||
!principal ||
|
||||
typeof principal !== 'object' ||
|
||||
typeof principal.userId !== 'string' ||
|
||||
!/^[A-Za-z0-9_-]{1,128}$/.test(principal.userId) ||
|
||||
typeof principal.accessToken !== 'string' ||
|
||||
!validJwt(principal.accessToken) ||
|
||||
typeof principal.expiresAt !== 'number' ||
|
||||
principal.expiresAt <= Date.now() ||
|
||||
principal.expiresAt > Date.now() + 610_000
|
||||
) {
|
||||
throw new Error('Orca smoke identity response principal is malformed')
|
||||
}
|
||||
return [name, principal]
|
||||
})
|
||||
)
|
||||
}
|
||||
|
||||
function parseRelayAsiaLoadPrincipals(value, expectedCount) {
|
||||
if (!Array.isArray(value) || value.length !== expectedCount) {
|
||||
throw new Error('Relay Asia load principal response is invalid')
|
||||
}
|
||||
const userIds = new Set()
|
||||
return value.map((principal, principalIndex) => {
|
||||
if (
|
||||
!principal || typeof principal !== 'object' ||
|
||||
principal.principalIndex !== principalIndex ||
|
||||
typeof principal.userId !== 'string' ||
|
||||
!/^usr_relay_asia_load_[A-Za-z0-9_-]{32}$/.test(principal.userId) ||
|
||||
typeof principal.profileId !== 'string' ||
|
||||
principal.profileId !== principal.userId.replace(/^usr_/, 'prof_') ||
|
||||
userIds.has(principal.userId) ||
|
||||
typeof principal.accessToken !== 'string' || !validJwt(principal.accessToken) ||
|
||||
typeof principal.expiresAt !== 'number' || principal.expiresAt <= Date.now() ||
|
||||
principal.expiresAt > Date.now() + 610_000
|
||||
) throw new Error('Relay Asia load principal response is malformed')
|
||||
userIds.add(principal.userId)
|
||||
return principal
|
||||
})
|
||||
}
|
||||
|
||||
async function readJson(response, label) {
|
||||
try {
|
||||
return await response.json()
|
||||
} catch {
|
||||
throw new Error(`${label} did not return JSON`)
|
||||
}
|
||||
}
|
||||
@@ -1,111 +0,0 @@
|
||||
import assert from 'node:assert/strict'
|
||||
import test from 'node:test'
|
||||
import { requestGitHubSmokeTokens } from './github-smoke-token.mjs'
|
||||
|
||||
const jwt = (value) => `${value}.${value}.${value}`
|
||||
|
||||
test('exchanges the runner OIDC token without returning request credentials', async () => {
|
||||
const requests = []
|
||||
const fetchImpl = async (url, init) => {
|
||||
requests.push({ url: String(url), init })
|
||||
if (requests.length === 1) return Response.json({ value: jwt('github') })
|
||||
return Response.json({
|
||||
accessTokens: Object.fromEntries(
|
||||
['owner', 'recipient', 'outsider'].map((name) => [
|
||||
name,
|
||||
{ userId: `usr_${name}`, accessToken: jwt(name), expiresAt: Date.now() + 600_000 }
|
||||
])
|
||||
)
|
||||
})
|
||||
}
|
||||
const result = await requestGitHubSmokeTokens(
|
||||
'https://auth-staging.onorca.dev',
|
||||
fetchImpl,
|
||||
{
|
||||
ACTIONS_ID_TOKEN_REQUEST_URL: 'https://actions.example.test/token?api-version=1',
|
||||
ACTIONS_ID_TOKEN_REQUEST_TOKEN: 'runner-request-token'
|
||||
}
|
||||
)
|
||||
assert.equal(result.owner.userId, 'usr_owner')
|
||||
assert.match(requests[0].url, /audience=https%3A%2F%2Fauth-staging\.onorca\.dev/)
|
||||
assert.equal(requests[0].init.headers.authorization, 'Bearer runner-request-token')
|
||||
assert.equal(requests[1].init.headers.authorization, `Bearer ${jwt('github')}`)
|
||||
})
|
||||
|
||||
test('fails with bounded errors and never includes credentials', async () => {
|
||||
await assert.rejects(
|
||||
requestGitHubSmokeTokens(
|
||||
'https://auth-staging.onorca.dev',
|
||||
async () => new Response('denied', { status: 403 }),
|
||||
{
|
||||
ACTIONS_ID_TOKEN_REQUEST_URL: 'https://actions.example.test/token',
|
||||
ACTIONS_ID_TOKEN_REQUEST_TOKEN: 'private-request-token'
|
||||
}
|
||||
),
|
||||
(error) => {
|
||||
assert.doesNotMatch(String(error), /private-request-token|denied/)
|
||||
return true
|
||||
}
|
||||
)
|
||||
})
|
||||
|
||||
test('requests and validates an exact Relay Asia principal batch', async () => {
|
||||
const requests = []
|
||||
const principals = Array.from({ length: 32 }, (_, principalIndex) => {
|
||||
const suffix = String(principalIndex).padStart(32, 'a')
|
||||
return {
|
||||
principalIndex,
|
||||
userId: `usr_relay_asia_load_${suffix}`,
|
||||
profileId: `prof_relay_asia_load_${suffix}`,
|
||||
accessToken: jwt(`load${principalIndex}`),
|
||||
expiresAt: Date.now() + 600_000
|
||||
}
|
||||
})
|
||||
const result = await requestGitHubSmokeTokens(
|
||||
'https://auth-staging.onorca.dev',
|
||||
async (url, init) => {
|
||||
requests.push({ url: String(url), init })
|
||||
return requests.length === 1
|
||||
? Response.json({ value: jwt('github') })
|
||||
: Response.json({
|
||||
accessTokens: Object.fromEntries(['owner', 'recipient', 'outsider'].map((name) => [
|
||||
name,
|
||||
{ userId: `usr_${name}`, accessToken: jwt(name), expiresAt: Date.now() + 600_000 }
|
||||
])),
|
||||
relayAsiaLoadPrincipals: principals
|
||||
})
|
||||
},
|
||||
{
|
||||
ACTIONS_ID_TOKEN_REQUEST_URL: 'https://actions.example.test/token',
|
||||
ACTIONS_ID_TOKEN_REQUEST_TOKEN: 'runner-request-token'
|
||||
},
|
||||
{ relayAsiaLoad: { shardIndex: 3, principalCount: 32 } }
|
||||
)
|
||||
assert.equal(result.relayAsiaLoadPrincipals.length, 32)
|
||||
assert.deepEqual(JSON.parse(requests[1].init.body), {
|
||||
relayAsiaLoad: { v: 1, shardIndex: 3, principalCount: 32 }
|
||||
})
|
||||
assert.equal(requests[1].init.headers['content-type'], 'application/json')
|
||||
})
|
||||
|
||||
test('rejects malformed or duplicate Relay Asia principal batches', async () => {
|
||||
const environment = {
|
||||
ACTIONS_ID_TOKEN_REQUEST_URL: 'https://actions.example.test/token',
|
||||
ACTIONS_ID_TOKEN_REQUEST_TOKEN: 'runner-request-token'
|
||||
}
|
||||
let request = 0
|
||||
await assert.rejects(requestGitHubSmokeTokens(
|
||||
'https://auth-staging.onorca.dev',
|
||||
async () => ++request === 1
|
||||
? Response.json({ value: jwt('github') })
|
||||
: Response.json({
|
||||
accessTokens: Object.fromEntries(['owner', 'recipient', 'outsider'].map((name) => [
|
||||
name,
|
||||
{ userId: `usr_${name}`, accessToken: jwt(name), expiresAt: Date.now() + 600_000 }
|
||||
])),
|
||||
relayAsiaLoadPrincipals: []
|
||||
}),
|
||||
environment,
|
||||
{ relayAsiaLoad: { shardIndex: 0, principalCount: 32 } }
|
||||
), /principal response is invalid/)
|
||||
})
|
||||
@@ -1,146 +0,0 @@
|
||||
import { execFileSync } from 'node:child_process'
|
||||
import { existsSync } from 'node:fs'
|
||||
import { homedir } from 'node:os'
|
||||
import { join } from 'node:path'
|
||||
import { assertStagingRelayAwake } from './staging-relay-apply-guard.mjs'
|
||||
|
||||
// The relay root keeps its historical path so every existing caller — 9 workflows, the fence
|
||||
// broker, and the infra:* scripts — is unchanged when --root is omitted. The foundation and apps
|
||||
// roots stay in the private repository with the services they own.
|
||||
const ROOT_DIRECTORIES = {
|
||||
relay: join('infra', 'terraform')
|
||||
}
|
||||
|
||||
const command = process.argv[2]
|
||||
const environment = readEnvironment(process.argv.slice(3))
|
||||
const root = readRoot(process.argv.slice(3))
|
||||
const tool = process.env.IAC_TOOL || findTool()
|
||||
|
||||
if (!command || !['init', 'plan', 'apply'].includes(command)) {
|
||||
exitWithUsage()
|
||||
}
|
||||
|
||||
if (!environment) {
|
||||
exitWithUsage('Missing --env staging|production')
|
||||
}
|
||||
|
||||
if (!root) {
|
||||
exitWithUsage(`Unknown --root; expected one of ${Object.keys(ROOT_DIRECTORIES).join('|')}`)
|
||||
}
|
||||
|
||||
const rootDirectory = ROOT_DIRECTORIES[root]
|
||||
const terraformDir = join(process.cwd(), rootDirectory)
|
||||
const backendConfig = join(terraformDir, 'backend', `${environment}.hcl`)
|
||||
const varFile = join(terraformDir, 'environments', `${environment}.tfvars`)
|
||||
|
||||
if (!existsSync(backendConfig)) {
|
||||
throw new Error(`Backend config not found: ${backendConfig}`)
|
||||
}
|
||||
|
||||
if (!existsSync(varFile)) {
|
||||
throw new Error(`Variable file not found: ${varFile}`)
|
||||
}
|
||||
|
||||
const chdir = `-chdir=${rootDirectory}`
|
||||
|
||||
if (command === 'init') {
|
||||
run([chdir, 'init', `-backend-config=backend/${environment}.hcl`])
|
||||
} else if (command === 'plan') {
|
||||
run([
|
||||
chdir,
|
||||
'plan',
|
||||
`-var-file=environments/${environment}.tfvars`,
|
||||
`-out=${environment}.tfplan`
|
||||
])
|
||||
} else {
|
||||
// A normal staging apply must not implicitly wake or partially mutate a sleeping data plane.
|
||||
// Only the relay root can touch that data plane; the guard would refuse app work for no reason.
|
||||
if (environment === 'staging' && root === 'relay') assertStagingRelayAwake()
|
||||
run([chdir, 'apply', `${environment}.tfplan`])
|
||||
}
|
||||
|
||||
function readEnvironment(args) {
|
||||
const envIndex = args.indexOf('--env')
|
||||
if (envIndex >= 0) {
|
||||
return args[envIndex + 1]
|
||||
}
|
||||
|
||||
return process.env.ORCA_CLOUD_ENV
|
||||
}
|
||||
|
||||
function readRoot(args) {
|
||||
const rootIndex = args.indexOf('--root')
|
||||
const requested = rootIndex >= 0 ? args[rootIndex + 1] : 'relay'
|
||||
return requested in ROOT_DIRECTORIES ? requested : undefined
|
||||
}
|
||||
|
||||
function findTool() {
|
||||
for (const candidate of ['tofu', 'terraform']) {
|
||||
try {
|
||||
execFileSync(candidate, ['version'], { stdio: 'ignore' })
|
||||
return candidate
|
||||
} catch {
|
||||
// Try the next compatible IaC binary.
|
||||
}
|
||||
}
|
||||
|
||||
throw new Error('Install Terraform or OpenTofu, or set IAC_TOOL.')
|
||||
}
|
||||
|
||||
function run(args) {
|
||||
execFileSync(tool, args, { env: terraformEnv(), stdio: 'inherit' })
|
||||
}
|
||||
|
||||
function terraformEnv() {
|
||||
if (
|
||||
process.env.GOOGLE_APPLICATION_CREDENTIALS ||
|
||||
process.env.GOOGLE_CREDENTIALS ||
|
||||
process.env.GOOGLE_OAUTH_ACCESS_TOKEN
|
||||
) {
|
||||
return process.env
|
||||
}
|
||||
|
||||
const token = readGcloudAccessToken()
|
||||
if (!token) {
|
||||
return process.env
|
||||
}
|
||||
|
||||
// Local convenience: Terraform uses ADC, while engineers often only have
|
||||
// gcloud CLI auth. CI should use Workload Identity instead.
|
||||
return { ...process.env, GOOGLE_OAUTH_ACCESS_TOKEN: token }
|
||||
}
|
||||
|
||||
function readGcloudAccessToken() {
|
||||
for (const candidate of gcloudCandidates()) {
|
||||
try {
|
||||
return execFileSync(candidate, ['auth', 'print-access-token'], {
|
||||
encoding: 'utf8',
|
||||
stdio: ['ignore', 'pipe', 'ignore']
|
||||
}).trim()
|
||||
} catch {
|
||||
// Try the next common gcloud location.
|
||||
}
|
||||
}
|
||||
|
||||
return null
|
||||
}
|
||||
|
||||
function gcloudCandidates() {
|
||||
return [
|
||||
process.env.GCLOUD_PATH,
|
||||
'gcloud',
|
||||
join(homedir(), 'Downloads', 'google-cloud-sdk', 'bin', 'gcloud'),
|
||||
join(homedir(), 'google-cloud-sdk', 'bin', 'gcloud')
|
||||
].filter(Boolean)
|
||||
}
|
||||
|
||||
function exitWithUsage(message) {
|
||||
if (message) {
|
||||
console.error(message)
|
||||
}
|
||||
|
||||
console.error(
|
||||
'Usage: pnpm infra:<init|plan|apply> --env staging|production [--root relay]'
|
||||
)
|
||||
process.exit(1)
|
||||
}
|
||||
@@ -1,77 +0,0 @@
|
||||
import assert from 'node:assert/strict'
|
||||
import { execFileSync } from 'node:child_process'
|
||||
import { readFileSync } from 'node:fs'
|
||||
import { test } from 'node:test'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
|
||||
const repository = fileURLToPath(new URL('../../', import.meta.url))
|
||||
const script = 'dev/scripts/infra.mjs'
|
||||
|
||||
// IAC_TOOL=echo prints the argv the real binary would have received, so the root a flag selects
|
||||
// is observable without running Terraform.
|
||||
function invoke(args) {
|
||||
return execFileSync('node', [script, ...args], {
|
||||
cwd: repository,
|
||||
encoding: 'utf8',
|
||||
env: { ...process.env, IAC_TOOL: 'echo' }
|
||||
}).trim()
|
||||
}
|
||||
|
||||
function rejects(args) {
|
||||
try {
|
||||
execFileSync('node', [script, ...args], {
|
||||
cwd: repository,
|
||||
encoding: 'utf8',
|
||||
stdio: ['ignore', 'pipe', 'pipe'],
|
||||
env: { ...process.env, IAC_TOOL: 'echo' }
|
||||
})
|
||||
} catch (error) {
|
||||
return error.stderr
|
||||
}
|
||||
throw new Error(`expected ${args.join(' ')} to exit non-zero`)
|
||||
}
|
||||
|
||||
// Why: 9 relay workflows, the fence broker, and the three infra:* package scripts all invoke this
|
||||
// without --root. If the default ever moves off infra/terraform they break silently at the plan.
|
||||
test('omitting --root keeps every existing caller on the relay root', () => {
|
||||
for (const environment of ['staging', 'production']) {
|
||||
assert.equal(
|
||||
invoke(['init', '--env', environment]),
|
||||
`-chdir=infra/terraform init -backend-config=backend/${environment}.hcl`
|
||||
)
|
||||
assert.equal(
|
||||
invoke(['plan', '--env', environment]),
|
||||
`-chdir=infra/terraform plan -var-file=environments/${environment}.tfvars -out=${environment}.tfplan`
|
||||
)
|
||||
}
|
||||
})
|
||||
|
||||
// Only the relay root ships here; the foundation and apps roots stay in the private repository.
|
||||
test('each root name selects exactly its own directory', () => {
|
||||
const directories = { relay: 'infra/terraform' }
|
||||
for (const [root, directory] of Object.entries(directories)) {
|
||||
for (const environment of ['staging', 'production']) {
|
||||
assert.equal(
|
||||
invoke(['init', '--env', environment, '--root', root]),
|
||||
`-chdir=${directory} init -backend-config=backend/${environment}.hcl`
|
||||
)
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
test('an unknown root fails closed rather than falling back to the relay root', () => {
|
||||
const stderr = rejects(['plan', '--env', 'staging', '--root', 'releay'])
|
||||
assert.match(stderr, /Unknown --root/)
|
||||
assert.doesNotMatch(stderr, /infra\/terraform /)
|
||||
})
|
||||
|
||||
test('a missing environment still fails before any root is resolved', () => {
|
||||
assert.match(rejects(['plan']), /Missing --env/)
|
||||
})
|
||||
|
||||
// Why: the guard refuses a staging apply while the relay data plane is asleep. Applying it to the
|
||||
// app or foundation roots would block work that never touches that data plane.
|
||||
test('the sleeping staging relay guard is scoped to the relay root', () => {
|
||||
const source = readFileSync(new URL('./infra.mjs', import.meta.url), 'utf8')
|
||||
assert.match(source, /environment === 'staging' && root === 'relay'/)
|
||||
})
|
||||
@@ -1,695 +0,0 @@
|
||||
import { createHash, createPrivateKey, createPublicKey } from 'node:crypto'
|
||||
import { readFileSync } from 'node:fs'
|
||||
import { monitorEventLoopDelay } from 'node:perf_hooks'
|
||||
import { setTimeout as delay } from 'node:timers/promises'
|
||||
import { RelayLoadControlPeer } from './relay-load-control-peer.mjs'
|
||||
import { RELAY_ASIA_LOAD_PRINCIPAL_SHARDS, requestGitHubSmokeTokens } from './github-smoke-token.mjs'
|
||||
import { relayLoadFailureReason } from './relay-load-connection-failure.mjs'
|
||||
import {
|
||||
assertRelayLoadDirectorCapacityToken,
|
||||
waitForRelayLoadDirectorCapacity,
|
||||
waitForRelayLoadRequestUnits
|
||||
} from './relay-load-director-capacity-gate.mjs'
|
||||
import { waitForRelayLoadPhaseBarrier } from './relay-load-phase-barrier.mjs'
|
||||
import {
|
||||
proveRelayLoadPlacementBoundary,
|
||||
proveRelayLoadRegionalFallback
|
||||
} from './relay-load-placement-boundary.mjs'
|
||||
import {
|
||||
proveRelayLoadRebindBoundary,
|
||||
waitForRelayLoadRebindGate
|
||||
} from './relay-load-rebind-boundary.mjs'
|
||||
import { proveRelayLoadRegionBehavior } from './relay-load-region-behavior.mjs'
|
||||
import { proveRelayLoadDuplicateAssign } from './relay-load-duplicate-assign.mjs'
|
||||
import {
|
||||
openRelayLoadInviteOffers,
|
||||
proveRelayLoadRequestUnitBoundary
|
||||
} from './relay-load-request-unit-boundary.mjs'
|
||||
import {
|
||||
assertRelayLoadRampAccepted,
|
||||
relayLoadRunHasDisallowedFailures,
|
||||
runRelayLoadWithShutdown
|
||||
} from './relay-load-run-lifecycle.mjs'
|
||||
import { createRelayLoadReaderEvidence } from './relay-load-reader-evidence.mjs'
|
||||
import { holdRelayLoadSplice, relayLoadSpliceReopenSummary } from './relay-load-splice-retry.mjs'
|
||||
import {
|
||||
installRelayLoadResumeDevice,
|
||||
relayLoadResumeSummary,
|
||||
resumeRelayLoadPhoneOnce
|
||||
} from './relay-load-resume-phone.mjs'
|
||||
import {
|
||||
parseRelayLoadArguments,
|
||||
relayLoadPrincipalIndex,
|
||||
relayLoadReaderEvidenceError,
|
||||
relayLoadResumePhoneIndexes,
|
||||
relayLoadSpliceIndexes,
|
||||
relayLoadSpliceProfile,
|
||||
relayLoadSpliceStartDelayMs
|
||||
} from './relay-load-profile.mjs'
|
||||
|
||||
function signingKey(path) {
|
||||
if (!path) return {}
|
||||
const key = createPrivateKey(readFileSync(path, 'utf8'))
|
||||
const signingKeyId = createHash('sha256')
|
||||
.update(createPublicKey(key).export({ type: 'spki', format: 'der' }))
|
||||
.digest('base64url')
|
||||
.slice(0, 16)
|
||||
return { signingKey: key, signingKeyId }
|
||||
}
|
||||
|
||||
function report(state, final = false) {
|
||||
const elapsedSeconds = Math.max(1, (Date.now() - state.startedAt) / 1000)
|
||||
const memory = process.memoryUsage()
|
||||
const cpu = process.cpuUsage(state.generatorBaselineCpu)
|
||||
const rssMiB = memory.rss / 1_048_576
|
||||
state.generatorPeakRssMiB = Math.max(state.generatorPeakRssMiB, rssMiB)
|
||||
const readerQueueEvidence = state.readerEvidence?.snapshot() ?? []
|
||||
const output = {
|
||||
event: final ? 'relay_load_complete' : 'relay_load_progress',
|
||||
controls: state.controls,
|
||||
shardCount: state.shardCount,
|
||||
shardIndex: state.shardIndex,
|
||||
configuredRampSeconds: state.rampMs / 1000,
|
||||
configuredSteadySeconds: state.durationMs / 1000,
|
||||
configuredSpliceHoldSeconds: state.spliceHoldMs / 1000,
|
||||
requiredLeaseHorizons: state.requiredLeaseHorizons,
|
||||
configuredSplices: state.splices,
|
||||
configuredSlowReaderSplices: state.slowReaderSplices,
|
||||
configuredWedgedReaderSplices: state.wedgedReaderSplices,
|
||||
active: state.active.size,
|
||||
peakActive: state.peakActive,
|
||||
steadyMinimumActive: state.steadyMinimumActive,
|
||||
connected: state.connected,
|
||||
connectionFailures: state.connectionFailures,
|
||||
rampConnectionFailures: state.rampConnectionFailures,
|
||||
steadyConnectionFailures: state.steadyConnectionFailures,
|
||||
transitionConnectionFailures: state.transitionConnectionFailures,
|
||||
connectionFailuresByReason: state.connectionFailuresByReason,
|
||||
closes: state.closes,
|
||||
unexpectedCloses: state.unexpectedCloses,
|
||||
unexpectedClosesByCode: state.unexpectedClosesByCode,
|
||||
drains: state.drains,
|
||||
pings: state.pings,
|
||||
pingRate: Number((state.pings / elapsedSeconds).toFixed(2)),
|
||||
tokens: state.tokens,
|
||||
tokenRate: Number((state.tokens / elapsedSeconds).toFixed(2)),
|
||||
refreshes: state.refreshes,
|
||||
refreshErrors: state.refreshErrors,
|
||||
protocolErrors: state.protocolErrors,
|
||||
socketErrors: state.socketErrors,
|
||||
rebindProbesOpened: state.rebindProbesOpened,
|
||||
rebindOverflowReason: state.rebindOverflowReason,
|
||||
placementOverflowReason: state.placementOverflowReason,
|
||||
regionalFallbacksProved: state.regionalFallbacksProved,
|
||||
oldClientUsFirstProved: state.oldClientUsFirstProved,
|
||||
stickyAssignmentProved: state.stickyAssignmentProved,
|
||||
duplicateAssign: state.duplicateAssign,
|
||||
requestUnitInvitesOpened: state.requestUnitInvitesOpened,
|
||||
requestUnitPrincipalCount: state.requestUnitPrincipalCount,
|
||||
relayAsiaLoadPrincipalCount: state.relayAsiaLoadPrincipalCount,
|
||||
requestUnitOverflowReason: state.requestUnitOverflowReason,
|
||||
requestUnitCleanupProved: state.requestUnitCleanupProved,
|
||||
phaseBarrierPassed: state.phaseBarrierPassed,
|
||||
activeSplices: state.activeSplices,
|
||||
peakActiveSplices: state.peakActiveSplices,
|
||||
completedSplices: state.completedSplices,
|
||||
failedSplices: state.failedSplices,
|
||||
spliceRetries: state.spliceRetries,
|
||||
spliceFailuresByReason: state.spliceFailuresByReason,
|
||||
spliceReopen: relayLoadSpliceReopenSummary(state.spliceReopenGapsMs, state.spliceFailedAt.size),
|
||||
slowReaderSplicesCompleted: state.slowReaderSplicesCompleted,
|
||||
wedgedReaderSplicesClosed: state.wedgedReaderSplicesClosed,
|
||||
readerQueueEvidence,
|
||||
readerQueuedBytesPeak: Math.max(
|
||||
0,
|
||||
...readerQueueEvidence.map(({ increaseBytes }) => increaseBytes)
|
||||
),
|
||||
readerClosesByCode: state.readerClosesByCode,
|
||||
resumePhones: relayLoadResumeSummary(state.resumeAttempts),
|
||||
// Every cell a control connected to, so a canary can prove its load reached the target.
|
||||
assignedCellOrigins: [...state.assignedCellOrigins].sort(),
|
||||
controlHeadroom: Math.max(0, state.controls - state.active.size),
|
||||
generatorRssMiB: Number(rssMiB.toFixed(1)),
|
||||
generatorPeakRssMiB: Number(state.generatorPeakRssMiB.toFixed(1)),
|
||||
generatorRssGrowthMiB: Number(
|
||||
Math.max(0, state.generatorPeakRssMiB - state.generatorBaselineRssMiB).toFixed(1)
|
||||
),
|
||||
generatorHeapUsedMiB: Number((memory.heapUsed / 1_048_576).toFixed(1)),
|
||||
generatorCpuPercent: Number(
|
||||
(((cpu.user + cpu.system) / 1_000_000 / elapsedSeconds) * 100).toFixed(1)
|
||||
),
|
||||
generatorEventLoopP99Ms: Number((state.eventLoopDelay.percentile(99) / 1_000_000).toFixed(2)),
|
||||
shutdownEvidence: final ? state.shutdownEvidence : undefined,
|
||||
elapsedSeconds: Number(elapsedSeconds.toFixed(1))
|
||||
}
|
||||
console.log(JSON.stringify(output))
|
||||
return output
|
||||
}
|
||||
|
||||
const config = parseRelayLoadArguments(process.argv.slice(2))
|
||||
let accessToken = process.env.ORCA_RELAY_LOAD_ACCESS_TOKEN
|
||||
let accessTokenProviderForIndex
|
||||
const adminToken = process.env.ORCA_RELAY_ADMIN_ID_TOKEN
|
||||
if (
|
||||
config.placementOverflowProbes > 0 || config.regionalFallbackProbes > 0 ||
|
||||
config.slowReaderSplices + config.wedgedReaderSplices > 0 ||
|
||||
config.requestUnitOverflowProbes > 0 || config.requestUnitCleanupTimeoutMs > 0
|
||||
) {
|
||||
assertRelayLoadDirectorCapacityToken({
|
||||
directorOrigin: config.directorOrigin,
|
||||
adminToken
|
||||
}, Date.now,
|
||||
config.rampMs + config.durationMs + config.wedgedReaderHoldMs +
|
||||
(config.phaseBarrierDir ? 2 * config.phaseBarrierTimeoutMs : 0) +
|
||||
config.spliceRampMs + config.requestUnitCleanupTimeoutMs + 120_000)
|
||||
}
|
||||
const key = signingKey(config.signingKeyFile)
|
||||
if (!accessToken && !key.signingKey && process.env.ACTIONS_ID_TOKEN_REQUEST_URL &&
|
||||
process.env.ACTIONS_ID_TOKEN_REQUEST_TOKEN) {
|
||||
let tokens
|
||||
let refresh
|
||||
const loadOptions = config.relayAsiaLoadPrincipalCount > 0
|
||||
? {
|
||||
relayAsiaLoad: {
|
||||
// Staging auth mints principal sets for shards 0-3 only; shards past that (a second
|
||||
// hosted runner) reuse a set, and their hosts stay distinct by peer index.
|
||||
shardIndex: config.shardIndex % RELAY_ASIA_LOAD_PRINCIPAL_SHARDS,
|
||||
principalCount: config.relayAsiaLoadPrincipalCount
|
||||
}
|
||||
}
|
||||
: undefined
|
||||
const smokeTokens = async () => {
|
||||
const expiresAt = config.relayAsiaLoadPrincipalCount > 0
|
||||
? tokens?.relayAsiaLoadPrincipals?.[0]?.expiresAt
|
||||
: tokens?.owner?.expiresAt
|
||||
if (expiresAt > Date.now() + 60_000) return tokens
|
||||
refresh ??= requestGitHubSmokeTokens(
|
||||
config.authOrigin,
|
||||
fetch,
|
||||
process.env,
|
||||
loadOptions
|
||||
)
|
||||
try {
|
||||
tokens = await refresh
|
||||
return tokens
|
||||
} finally {
|
||||
refresh = undefined
|
||||
}
|
||||
}
|
||||
accessTokenProviderForIndex = (index) => async () => {
|
||||
const current = await smokeTokens()
|
||||
return config.relayAsiaLoadPrincipalCount > 0
|
||||
? current.relayAsiaLoadPrincipals[
|
||||
relayLoadPrincipalIndex(
|
||||
index,
|
||||
config.shardCount,
|
||||
current.relayAsiaLoadPrincipals.length
|
||||
)
|
||||
].accessToken
|
||||
: current.owner.accessToken
|
||||
}
|
||||
await smokeTokens()
|
||||
}
|
||||
if (!accessToken && !accessTokenProviderForIndex && !key.signingKey) {
|
||||
throw new Error('provide GitHub OIDC, ORCA_RELAY_LOAD_ACCESS_TOKEN, or --signing-key-file')
|
||||
}
|
||||
const eventLoopDelay = monitorEventLoopDelay({ resolution: 20 })
|
||||
eventLoopDelay.enable()
|
||||
const generatorBaselineRssMiB = process.memoryUsage().rss / 1_048_576
|
||||
const generatorBaselineCpu = process.cpuUsage()
|
||||
const state = {
|
||||
...config,
|
||||
startedAt: Date.now(),
|
||||
active: new Set(),
|
||||
peakActive: 0,
|
||||
steadyMinimumActive: null,
|
||||
steadyStarted: false,
|
||||
connected: 0,
|
||||
connectionFailures: 0,
|
||||
rampConnectionFailures: 0,
|
||||
steadyConnectionFailures: 0,
|
||||
transitionConnectionFailures: 0,
|
||||
connectionFailuresByReason: {},
|
||||
closes: 0,
|
||||
unexpectedCloses: 0,
|
||||
unexpectedClosesByCode: {},
|
||||
drains: 0,
|
||||
pings: 0,
|
||||
tokens: 0,
|
||||
refreshes: 0,
|
||||
refreshErrors: 0,
|
||||
protocolErrors: 0,
|
||||
socketErrors: 0,
|
||||
rebindProbesOpened: 0,
|
||||
rebindOverflowReason: null,
|
||||
placementOverflowReason: null,
|
||||
regionalFallbacksProved: 0,
|
||||
oldClientUsFirstProved: 0,
|
||||
stickyAssignmentProved: 0,
|
||||
duplicateAssign: null,
|
||||
requestUnitInvitesOpened: 0,
|
||||
requestUnitOverflowReason: null,
|
||||
requestUnitCleanupProved: 0,
|
||||
phaseBarrierPassed: false,
|
||||
activeSplices: 0,
|
||||
peakActiveSplices: 0,
|
||||
completedSplices: 0,
|
||||
failedSplices: 0,
|
||||
spliceRetries: 0,
|
||||
spliceFailuresByReason: {},
|
||||
spliceFailedAt: new Map(),
|
||||
resumeAttempts: [],
|
||||
spliceReopenGapsMs: [],
|
||||
slowReaderSplicesCompleted: 0,
|
||||
wedgedReaderSplicesClosed: 0,
|
||||
readerEvidence: null,
|
||||
readerClosesByCode: {},
|
||||
assignedCellOrigins: new Set(),
|
||||
generatorBaselineRssMiB,
|
||||
generatorBaselineCpu,
|
||||
generatorPeakRssMiB: generatorBaselineRssMiB,
|
||||
peerShutdowns: 0,
|
||||
shutdownEvidence: null,
|
||||
eventLoopDelay,
|
||||
stopping: false,
|
||||
transitionWindow: false
|
||||
}
|
||||
const peers = new Map()
|
||||
const reconnectTimers = new Set()
|
||||
// Duplicate-assign probe hosts: outside `peers`, so they never count toward the run's controls.
|
||||
const probePeers = []
|
||||
|
||||
async function readRuntimeQueuedBytes(origin) {
|
||||
const response = await fetch(`${origin}/v1/admin/runtime-status`, {
|
||||
method: 'POST',
|
||||
headers: { authorization: `Bearer ${adminToken}`, 'content-type': 'application/json' },
|
||||
body: JSON.stringify({ v: 1 }),
|
||||
signal: AbortSignal.timeout(5_000)
|
||||
})
|
||||
if (response.status === 401 || response.status === 403) {
|
||||
throw new Error('reader evidence identity was rejected')
|
||||
}
|
||||
if (!response.ok) throw new Error(`reader runtime status returned ${response.status}`)
|
||||
const status = await response.json()
|
||||
const queuedBytes = status?.runtime?.queuedBytes
|
||||
if (!Number.isSafeInteger(queuedBytes) || queuedBytes < 0) {
|
||||
throw new Error('reader runtime queued bytes are invalid')
|
||||
}
|
||||
return queuedBytes
|
||||
}
|
||||
|
||||
async function observeReaderPressure(input) {
|
||||
if (!state.readerEvidence) throw new Error('reader evidence baseline is unavailable')
|
||||
await state.readerEvidence.observe(input)
|
||||
state.generatorPeakRssMiB = Math.max(
|
||||
state.generatorPeakRssMiB,
|
||||
process.memoryUsage().rss / 1_048_576
|
||||
)
|
||||
}
|
||||
|
||||
function recordSteadyMinimum() {
|
||||
if (!state.steadyStarted || state.stopping) return
|
||||
state.steadyMinimumActive = Math.min(state.steadyMinimumActive, state.active.size)
|
||||
}
|
||||
|
||||
function scheduleReconnect(peer) {
|
||||
if (state.stopping) return
|
||||
const timeout = setTimeout(() => {
|
||||
reconnectTimers.delete(timeout)
|
||||
void connect(peer)
|
||||
}, Math.floor(Math.random() * (config.reconnectMaxMs + 1)))
|
||||
reconnectTimers.add(timeout)
|
||||
}
|
||||
|
||||
function observe(type, detail) {
|
||||
if (type === 'connected') {
|
||||
state.active.add(detail.index)
|
||||
state.connected++
|
||||
const cellOrigin = peers.get(detail.index)?.assignedCellUrl()
|
||||
if (cellOrigin) state.assignedCellOrigins.add(cellOrigin)
|
||||
state.peakActive = Math.max(state.peakActive, state.active.size)
|
||||
recordSteadyMinimum()
|
||||
} else if (type === 'closed') {
|
||||
state.active.delete(detail.index)
|
||||
state.closes++
|
||||
if (!detail.stopped && !detail.expectedDrain) {
|
||||
state.unexpectedCloses++
|
||||
const code = String(detail.code)
|
||||
state.unexpectedClosesByCode[code] = (state.unexpectedClosesByCode[code] ?? 0) + 1
|
||||
}
|
||||
if (!detail.stopped) scheduleReconnect(peers.get(detail.index))
|
||||
recordSteadyMinimum()
|
||||
} else if (type === 'drain') state.drains++
|
||||
else if (type === 'ping') state.pings++
|
||||
else if (type === 'token') state.tokens++
|
||||
else if (type === 'refresh') state.refreshes++
|
||||
else if (type === 'refreshError') state.refreshErrors++
|
||||
else if (type === 'protocolError') state.protocolErrors++
|
||||
else if (type === 'socketError') state.socketErrors++
|
||||
else if (type === 'spliceOpened') {
|
||||
const failedAt = state.spliceFailedAt.get(detail.index)
|
||||
if (failedAt !== undefined) {
|
||||
state.spliceReopenGapsMs.push(Date.now() - failedAt)
|
||||
state.spliceFailedAt.delete(detail.index)
|
||||
}
|
||||
state.activeSplices++
|
||||
state.peakActiveSplices = Math.max(state.peakActiveSplices, state.activeSplices)
|
||||
} else if (type === 'spliceCompleted') {
|
||||
state.completedSplices++
|
||||
if (detail.readerMode === 'slow') state.slowReaderSplicesCompleted++
|
||||
} else if (type === 'spliceWedged') {
|
||||
state.wedgedReaderSplicesClosed++
|
||||
const code = String(detail.code)
|
||||
state.readerClosesByCode[code] = (state.readerClosesByCode[code] ?? 0) + 1
|
||||
} else if (type === 'spliceClosed') state.activeSplices--
|
||||
else if (type === 'spliceFailed') state.failedSplices++
|
||||
else if (type === 'shutdown') state.peerShutdowns++
|
||||
}
|
||||
|
||||
async function connect(peer) {
|
||||
try {
|
||||
await peer.connect()
|
||||
} catch (error) {
|
||||
state.connectionFailures++
|
||||
if (state.steadyStarted) state.steadyConnectionFailures++
|
||||
else if (state.transitionWindow) state.transitionConnectionFailures++
|
||||
else state.rampConnectionFailures++
|
||||
const reason = relayLoadFailureReason(error)
|
||||
state.connectionFailuresByReason[reason] =
|
||||
(state.connectionFailuresByReason[reason] ?? 0) + 1
|
||||
scheduleReconnect(peer)
|
||||
}
|
||||
}
|
||||
|
||||
const peerOptions = (index, overrides = {}) => ({
|
||||
...config,
|
||||
...key,
|
||||
accessToken,
|
||||
...(accessTokenProviderForIndex
|
||||
? { accessTokenProvider: accessTokenProviderForIndex(index) }
|
||||
: {}),
|
||||
seed: 0x4f524341 ^ config.shardIndex,
|
||||
...overrides
|
||||
})
|
||||
if (config.regionBehaviorProbes > 0) {
|
||||
const proofIndex = config.controls * config.shardCount + 10_000
|
||||
const regionProof = await proveRelayLoadRegionBehavior({
|
||||
oldClientPeer: new RelayLoadControlPeer(
|
||||
proofIndex,
|
||||
peerOptions(proofIndex, { preferredRegion: undefined }),
|
||||
() => undefined
|
||||
),
|
||||
stickyPeer: new RelayLoadControlPeer(
|
||||
proofIndex + 1,
|
||||
peerOptions(proofIndex + 1, { preferredRegion: 'asia-east2' }),
|
||||
() => undefined
|
||||
),
|
||||
asiaOrigin: config.capacityCellOrigin
|
||||
})
|
||||
state.oldClientUsFirstProved = regionProof.oldClientUsFirst ? 1 : 0
|
||||
state.stickyAssignmentProved = regionProof.stickyAssignmentPreserved ? 1 : 0
|
||||
}
|
||||
const initialConnections = []
|
||||
for (let localIndex = 0; localIndex < config.controls; localIndex++) {
|
||||
const globalIndex = localIndex * config.shardCount + config.shardIndex
|
||||
const peer = new RelayLoadControlPeer(globalIndex, peerOptions(globalIndex), observe)
|
||||
peers.set(globalIndex, peer)
|
||||
const rampOffset =
|
||||
config.controls === 1 ? 0 : Math.floor((localIndex / (config.controls - 1)) * config.rampMs)
|
||||
const offset = config.rampStartDelayMs + rampOffset
|
||||
initialConnections.push(delay(offset).then(() => connect(peer)))
|
||||
}
|
||||
function recordResumeAttempt(peer, attempt) {
|
||||
state.resumeAttempts.push(attempt)
|
||||
console.log(JSON.stringify({ event: 'relay_load_resume_attempt', index: peer.index, ...attempt }))
|
||||
}
|
||||
// A paired phone that reconnects every interval until the run stops; every attempt is a line.
|
||||
async function runResumePhone(peer, localIndex, endsAt) {
|
||||
const relayDeviceId = `load-resume-device-${peer.index}`
|
||||
// Spread the phones across one interval.
|
||||
await delay(Math.floor((localIndex / Math.max(1, config.resumePhones)) * config.resumeIntervalMs))
|
||||
let token
|
||||
while (!state.stopping && Date.now() < endsAt) {
|
||||
const startedAt = Date.now()
|
||||
if (!token) {
|
||||
try {
|
||||
token = await installRelayLoadResumeDevice(peer, relayDeviceId)
|
||||
} catch (error) {
|
||||
recordResumeAttempt(peer, {
|
||||
at: startedAt,
|
||||
ok: false,
|
||||
failure: 'install',
|
||||
error: String(error.message).slice(0, 160)
|
||||
})
|
||||
}
|
||||
} else {
|
||||
recordResumeAttempt(peer, await resumeRelayLoadPhoneOnce(peer, { token, relayDeviceId }, {
|
||||
holdMs: config.resumeHoldMs,
|
||||
stopping: () => state.stopping,
|
||||
delay
|
||||
}))
|
||||
}
|
||||
await delay(Math.min(
|
||||
Math.max(1_000, config.resumeIntervalMs - (Date.now() - startedAt)),
|
||||
Math.max(0, endsAt - Date.now())
|
||||
))
|
||||
}
|
||||
}
|
||||
const progressTimer = setInterval(() => report(state), 10_000)
|
||||
progressTimer.unref()
|
||||
await runRelayLoadWithShutdown(async () => {
|
||||
await Promise.all(initialConnections)
|
||||
assertRelayLoadRampAccepted(state.rampConnectionFailures, config.maxRampConnectionFailures)
|
||||
if (config.duplicateAssignProbes > 0) {
|
||||
const reconnects = { closes: 0, closesByCode: {}, reconnectFailures: 0 }
|
||||
// A probe host closed by a demotion re-assigns through the director, as a desktop would.
|
||||
const observeProbe = (peer) => (type, detail) => {
|
||||
if (type !== 'closed' || detail.stopped) return
|
||||
reconnects.closes++
|
||||
const code = String(detail.code)
|
||||
reconnects.closesByCode[code] = (reconnects.closesByCode[code] ?? 0) + 1
|
||||
if (state.stopping) return
|
||||
const timeout = setTimeout(() => {
|
||||
reconnectTimers.delete(timeout)
|
||||
void peer.connect().catch(() => {
|
||||
reconnects.reconnectFailures++
|
||||
})
|
||||
}, 1_000 + Math.floor(Math.random() * 5_000))
|
||||
reconnectTimers.add(timeout)
|
||||
}
|
||||
const base = config.controls * config.shardCount + 20_000
|
||||
const pairs = Array.from({ length: config.duplicateAssignProbes }, (_, probe) => {
|
||||
const index = base + probe * config.shardCount + config.shardIndex
|
||||
const primary = new RelayLoadControlPeer(index, peerOptions(index), (type, detail) =>
|
||||
observeProbe(primary)(type, detail)
|
||||
)
|
||||
const twin = new RelayLoadControlPeer(
|
||||
index,
|
||||
peerOptions(index, { keys: primary.keys }),
|
||||
() => undefined
|
||||
)
|
||||
probePeers.push(primary, twin)
|
||||
return { primary, twin }
|
||||
})
|
||||
state.duplicateAssign = {
|
||||
...(await proveRelayLoadDuplicateAssign({
|
||||
pairs,
|
||||
holdMs: config.duplicateAssignHoldMs,
|
||||
delay,
|
||||
failureReason: relayLoadFailureReason
|
||||
})),
|
||||
probeReconnects: reconnects
|
||||
}
|
||||
}
|
||||
if (
|
||||
config.rebindProbes > 0 || config.placementOverflowProbes > 0 ||
|
||||
config.regionalFallbackProbes > 0
|
||||
) {
|
||||
state.transitionWindow = config.rebindDelayMs > 0
|
||||
await waitForRelayLoadRebindGate({
|
||||
delay,
|
||||
delayMs: config.rebindDelayMs,
|
||||
activeCount: () => state.active.size,
|
||||
requiredCount: config.controls
|
||||
})
|
||||
state.transitionWindow = false
|
||||
}
|
||||
if (config.placementOverflowProbes > 0 || config.regionalFallbackProbes > 0) {
|
||||
const closesBeforeBoundary = state.closes
|
||||
await waitForRelayLoadDirectorCapacity({
|
||||
directorOrigin: config.directorOrigin,
|
||||
adminToken,
|
||||
cellId: config.capacityCellId,
|
||||
hardCap: config.capacityHardCap,
|
||||
unobservedBound: config.capacityUnobservedBound,
|
||||
requiredConnections: config.aggregateControls
|
||||
})
|
||||
if (state.active.size !== config.controls || state.closes !== closesBeforeBoundary) {
|
||||
throw new Error('ordinary controls changed during the director capacity gate')
|
||||
}
|
||||
const overflowIndex = config.controls * config.shardCount + config.shardIndex
|
||||
if (config.placementOverflowProbes > 0) {
|
||||
state.placementOverflowReason = await proveRelayLoadPlacementBoundary({
|
||||
peer: new RelayLoadControlPeer(overflowIndex, peerOptions(overflowIndex), observe),
|
||||
failureReason: relayLoadFailureReason
|
||||
})
|
||||
}
|
||||
if (config.regionalFallbackProbes > 0) {
|
||||
await proveRelayLoadRegionalFallback({
|
||||
peer: new RelayLoadControlPeer(overflowIndex, peerOptions(overflowIndex), () => undefined),
|
||||
blockedOrigin: config.capacityCellOrigin
|
||||
})
|
||||
state.regionalFallbacksProved = 1
|
||||
}
|
||||
if (state.active.size !== config.controls || state.closes !== closesBeforeBoundary) {
|
||||
throw new Error('ordinary controls changed during the placement boundary probe')
|
||||
}
|
||||
await waitForRelayLoadDirectorCapacity({
|
||||
directorOrigin: config.directorOrigin,
|
||||
adminToken,
|
||||
cellId: config.capacityCellId,
|
||||
hardCap: config.capacityHardCap,
|
||||
unobservedBound: config.capacityUnobservedBound,
|
||||
requiredConnections: config.aggregateControls,
|
||||
requiredSamples: 1
|
||||
})
|
||||
if (state.active.size !== config.controls || state.closes !== closesBeforeBoundary) {
|
||||
throw new Error('ordinary controls changed before post-probe capacity verification')
|
||||
}
|
||||
}
|
||||
const rebindResult = await proveRelayLoadRebindBoundary({
|
||||
peers: [...state.active].map((index) => peers.get(index)),
|
||||
probeCount: config.rebindProbes,
|
||||
holdMs: config.rebindHoldMs,
|
||||
delay,
|
||||
failureReason: relayLoadFailureReason,
|
||||
requireOverflow: config.requireRebindOverflow
|
||||
})
|
||||
state.rebindProbesOpened = rebindResult.opened
|
||||
state.rebindOverflowReason = rebindResult.overflowReason
|
||||
if (config.requestUnitInvites > 0) {
|
||||
state.requestUnitInvitesOpened = await openRelayLoadInviteOffers({
|
||||
peers: [...state.active].sort((left, right) => left - right).map((index) => peers.get(index)),
|
||||
count: config.requestUnitInvites,
|
||||
ratePerSecond: config.requestUnitInvitesPerSecond
|
||||
})
|
||||
}
|
||||
if (config.requestUnitOverflowProbes > 0) {
|
||||
await waitForRelayLoadRequestUnits({
|
||||
directorOrigin: config.directorOrigin,
|
||||
adminToken,
|
||||
cellId: config.capacityCellId,
|
||||
capacityRequests: config.requestUnitCapacity,
|
||||
expectedRequestUnits: config.requestUnitCapacity,
|
||||
expectedActivityLeases: config.requestUnitCapacity
|
||||
})
|
||||
state.requestUnitOverflowReason = await proveRelayLoadRequestUnitBoundary(
|
||||
peers.get([...state.active][0])
|
||||
)
|
||||
}
|
||||
if (config.phaseBarrierDir) {
|
||||
await waitForRelayLoadPhaseBarrier({
|
||||
directory: config.phaseBarrierDir,
|
||||
shardCount: config.shardCount,
|
||||
shardIndex: config.shardIndex,
|
||||
timeoutMs: config.phaseBarrierTimeoutMs
|
||||
})
|
||||
state.phaseBarrierPassed = true
|
||||
}
|
||||
state.steadyStarted = true
|
||||
state.steadyMinimumActive = state.active.size
|
||||
const spliceIndexes = relayLoadSpliceIndexes(config)
|
||||
const readerOrigins = spliceIndexes.flatMap((index, spliceIndex) =>
|
||||
relayLoadSpliceProfile(config, spliceIndex).readerMode === 'normal'
|
||||
? []
|
||||
: [peers.get(index).lastAssignment.cellUrl]
|
||||
)
|
||||
state.readerEvidence = await createRelayLoadReaderEvidence(readerOrigins, {
|
||||
readQueuedBytes: readRuntimeQueuedBytes,
|
||||
delay
|
||||
})
|
||||
if (config.phaseBarrierDir) {
|
||||
await waitForRelayLoadPhaseBarrier({
|
||||
directory: `${config.phaseBarrierDir}-splices`,
|
||||
shardCount: config.shardCount,
|
||||
shardIndex: config.shardIndex,
|
||||
timeoutMs: config.phaseBarrierTimeoutMs
|
||||
})
|
||||
}
|
||||
const splicePromises = spliceIndexes.map((index, spliceIndex) =>
|
||||
delay(relayLoadSpliceStartDelayMs(config, spliceIndex)).then(() =>
|
||||
holdRelayLoadSplice({
|
||||
open: () =>
|
||||
peers.get(index).openSplice({
|
||||
payloadBytes: config.splicePayloadBytes,
|
||||
...relayLoadSpliceProfile(config, spliceIndex),
|
||||
observeReaderPressure,
|
||||
holdMs: config.spliceHoldMs
|
||||
}),
|
||||
retryMaxMs: config.spliceRetryMaxMs,
|
||||
stopping: () => state.stopping,
|
||||
onFailure: (reason) => {
|
||||
state.spliceRetries++
|
||||
state.spliceFailuresByReason[reason] = (state.spliceFailuresByReason[reason] ?? 0) + 1
|
||||
if (!state.spliceFailedAt.has(index)) state.spliceFailedAt.set(index, Date.now())
|
||||
},
|
||||
delay
|
||||
})
|
||||
)
|
||||
)
|
||||
const resumeEndsAt = Date.now() + config.durationMs
|
||||
const resumePromises = relayLoadResumePhoneIndexes(config).map((index, localIndex) =>
|
||||
runResumePhone(peers.get(index), localIndex, resumeEndsAt)
|
||||
)
|
||||
await Promise.all([...splicePromises, ...resumePromises, delay(config.durationMs)])
|
||||
}, async () => {
|
||||
state.stopping = true
|
||||
clearInterval(progressTimer)
|
||||
for (const timeout of reconnectTimers) clearTimeout(timeout)
|
||||
reconnectTimers.clear()
|
||||
await Promise.all([...peers.values(), ...probePeers].map((peer) => peer.shutdown()))
|
||||
eventLoopDelay.disable()
|
||||
state.shutdownEvidence = {
|
||||
peerShutdowns: state.peerShutdowns,
|
||||
activeControls: state.active.size,
|
||||
activeSplices: state.activeSplices,
|
||||
reconnectTimers: reconnectTimers.size
|
||||
}
|
||||
})
|
||||
if (config.requestUnitCleanupTimeoutMs > 0) {
|
||||
await waitForRelayLoadRequestUnits({
|
||||
directorOrigin: config.directorOrigin,
|
||||
adminToken,
|
||||
cellId: config.capacityCellId,
|
||||
capacityRequests: config.requestUnitCapacity,
|
||||
expectedRequestUnits: 0,
|
||||
expectedActivityLeases: 0,
|
||||
timeoutMs: config.requestUnitCleanupTimeoutMs
|
||||
})
|
||||
state.requestUnitCleanupProved = 1
|
||||
}
|
||||
const result = report(state, true)
|
||||
const minimumPeak = config.allowPartial ? 1 : Math.ceil(config.controls * 0.95)
|
||||
if (result.peakActive < minimumPeak) {
|
||||
throw new Error(`peak active controls ${result.peakActive} below required ${minimumPeak}`)
|
||||
}
|
||||
if (result.steadyMinimumActive < minimumPeak) {
|
||||
throw new Error(
|
||||
`steady minimum active controls ${result.steadyMinimumActive} below required ${minimumPeak}`
|
||||
)
|
||||
}
|
||||
if (relayLoadRunHasDisallowedFailures(result, config)) {
|
||||
throw new Error('relay load run observed connection, protocol, refresh, or socket errors')
|
||||
}
|
||||
const readerEvidenceError = relayLoadReaderEvidenceError(result, config)
|
||||
if (readerEvidenceError) throw new Error(readerEvidenceError)
|
||||
if (
|
||||
result.failedSplices > 0 ||
|
||||
result.completedSplices + result.wedgedReaderSplicesClosed !== config.splices ||
|
||||
result.shutdownEvidence.peerShutdowns !== config.controls ||
|
||||
result.shutdownEvidence.activeControls !== 0 ||
|
||||
result.shutdownEvidence.activeSplices !== 0 ||
|
||||
result.shutdownEvidence.reconnectTimers !== 0
|
||||
) {
|
||||
throw new Error('relay load run did not complete splices or shut down cleanly')
|
||||
}
|
||||
@@ -1,161 +0,0 @@
|
||||
import { readFileSync } from 'node:fs'
|
||||
import { pathToFileURL } from 'node:url'
|
||||
|
||||
// Per-desktop disconnect gap for one drained cell, read from log lines that already ship:
|
||||
// the source cell's `control closed` line and the director's reconnect `assignment granted`
|
||||
// line (drains send `reconnect: true`, so every drained desktop's grant carries `hinted=true`).
|
||||
// Read-only: the operator exports both logs; this file never queries anything.
|
||||
|
||||
const CLOSE = /^\[orca-relay\] control closed host=(\S+) .*?\bsplices=(\d+) .*?\bcode=(\d+) reason=("(?:[^"\\]|\\.)*")/
|
||||
const GRANT = /^\[orca-relay\] assignment granted lane=\S+ hinted=true host=(\S+) cell=(\S+)$/
|
||||
// The cell's own drain close; a desktop closed this way had not moved yet, idle or not.
|
||||
const DRAIN_CLOSE_REASON = 'resolve configured director'
|
||||
|
||||
function entryText(entry) {
|
||||
if (typeof entry.textPayload === 'string') return entry.textPayload
|
||||
if (typeof entry.jsonPayload?.message === 'string') return entry.jsonPayload.message
|
||||
return null
|
||||
}
|
||||
|
||||
function entryTime(entry) {
|
||||
const at = Date.parse(entry.timestamp)
|
||||
if (!Number.isFinite(at)) throw new Error('log entry has no timestamp')
|
||||
return at
|
||||
}
|
||||
|
||||
export function readDrainCloses(entries) {
|
||||
const closes = []
|
||||
for (const entry of entries) {
|
||||
const match = CLOSE.exec(entryText(entry) ?? '')
|
||||
if (!match) continue
|
||||
closes.push({
|
||||
host: match[1],
|
||||
splices: Number(match[2]),
|
||||
code: Number(match[3]),
|
||||
reason: JSON.parse(match[4]),
|
||||
at: entryTime(entry)
|
||||
})
|
||||
}
|
||||
// gcloud exports newest first; the measurement needs each host's earliest close.
|
||||
return closes.sort((left, right) => left.at - right.at)
|
||||
}
|
||||
|
||||
export function readReconnectGrants(entries) {
|
||||
const grants = []
|
||||
for (const entry of entries) {
|
||||
const match = GRANT.exec(entryText(entry) ?? '')
|
||||
if (match) grants.push({ host: match[1], cellId: match[2], at: entryTime(entry) })
|
||||
}
|
||||
return grants.sort((left, right) => left.at - right.at)
|
||||
}
|
||||
|
||||
function percentile(sorted, fraction) {
|
||||
if (sorted.length === 0) return null
|
||||
return sorted[Math.min(sorted.length - 1, Math.ceil(fraction * sorted.length) - 1)]
|
||||
}
|
||||
|
||||
function summary(values) {
|
||||
const sorted = [...values].sort((left, right) => left - right)
|
||||
return {
|
||||
count: sorted.length,
|
||||
p50Ms: percentile(sorted, 0.5),
|
||||
p95Ms: percentile(sorted, 0.95),
|
||||
maxMs: sorted.at(-1) ?? null
|
||||
}
|
||||
}
|
||||
|
||||
// `controlsAtDrainStart` is the denominator: hosts the cell held when the drain began,
|
||||
// read from its runtime metrics line, so a desktop that never logged a close still counts.
|
||||
// `drainEndedAt` closes the window: after it the rolled cell's new container serves new sessions,
|
||||
// and their closes are not part of the drain. Grants after it still count, as the gap's far end.
|
||||
export function measureDrainDisconnectGap({
|
||||
sourceCellId,
|
||||
drainStartedAt,
|
||||
drainEndedAt,
|
||||
controlsAtDrainStart,
|
||||
closes,
|
||||
grants,
|
||||
cellRegions = {}
|
||||
}) {
|
||||
if (!Number.isFinite(drainStartedAt)) throw new Error('drain start time is invalid')
|
||||
if (!Number.isFinite(drainEndedAt) || drainEndedAt <= drainStartedAt) {
|
||||
throw new Error('drain end time is invalid')
|
||||
}
|
||||
const sourceRegion = cellRegions[sourceCellId]
|
||||
// First close per host after the drain began; later closes are the host's new sessions.
|
||||
const firstClose = new Map()
|
||||
for (const close of closes) {
|
||||
if (close.at < drainStartedAt || close.at > drainEndedAt || firstClose.has(close.host)) continue
|
||||
firstClose.set(close.host, close)
|
||||
}
|
||||
const grantsByHost = new Map()
|
||||
for (const grant of grants) {
|
||||
if (grant.at < drainStartedAt || grant.cellId === sourceCellId) continue
|
||||
grantsByHost.set(grant.host, [...(grantsByHost.get(grant.host) ?? []), grant])
|
||||
}
|
||||
const cutOffGapsMs = []
|
||||
const counts = { movedFirst: 0, cutOff: 0, cutOffUnresolved: 0, otherClose: 0 }
|
||||
const leftRegion = []
|
||||
let phoneSessionsDropped = 0
|
||||
for (const close of firstClose.values()) {
|
||||
phoneSessionsDropped += close.splices
|
||||
const hostGrants = grantsByHost.get(close.host) ?? []
|
||||
const first = hostGrants[0]
|
||||
if (first && first.at <= close.at) counts.movedFirst += 1
|
||||
else if (close.reason === DRAIN_CLOSE_REASON) {
|
||||
if (first) {
|
||||
counts.cutOff += 1
|
||||
cutOffGapsMs.push(first.at - close.at)
|
||||
} else counts.cutOffUnresolved += 1
|
||||
} else counts.otherClose += 1
|
||||
if (first && sourceRegion && cellRegions[first.cellId] && cellRegions[first.cellId] !== sourceRegion) {
|
||||
const back = hostGrants.find(
|
||||
(grant) => grant.at > first.at && cellRegions[grant.cellId] === sourceRegion
|
||||
)
|
||||
leftRegion.push(back ? back.at - first.at : null)
|
||||
}
|
||||
}
|
||||
const returned = leftRegion.filter((value) => value !== null)
|
||||
return {
|
||||
sourceCellId,
|
||||
controlsAtDrainStart,
|
||||
closedHosts: firstClose.size,
|
||||
// Below 1 means some drained desktops left no close line in the export; widen it.
|
||||
closeCoverage: controlsAtDrainStart > 0 ? firstClose.size / controlsAtDrainStart : null,
|
||||
...counts,
|
||||
// Lower bound until the target cells run an image that logs control activation.
|
||||
cutOffGap: summary(cutOffGapsMs),
|
||||
phoneSessionsDropped,
|
||||
leftSourceRegion: leftRegion.length,
|
||||
leftSourceRegionStillAway: leftRegion.length - returned.length,
|
||||
timeUntilBack: summary(returned)
|
||||
}
|
||||
}
|
||||
|
||||
function argument(name) {
|
||||
const index = process.argv.indexOf(`--${name}`)
|
||||
return index === -1 ? undefined : process.argv[index + 1]
|
||||
}
|
||||
|
||||
function required(name) {
|
||||
const value = argument(name)
|
||||
if (value === undefined) throw new Error(`--${name} is required`)
|
||||
return value
|
||||
}
|
||||
|
||||
function main() {
|
||||
const readJson = (path) => JSON.parse(readFileSync(path, 'utf8'))
|
||||
const cellRegionsPath = argument('cell-regions')
|
||||
const result = measureDrainDisconnectGap({
|
||||
sourceCellId: required('source-cell'),
|
||||
drainStartedAt: Date.parse(required('drain-started-at')),
|
||||
drainEndedAt: Date.parse(required('drain-ended-at')),
|
||||
controlsAtDrainStart: Number(required('controls')),
|
||||
closes: readDrainCloses(readJson(required('cell-log'))),
|
||||
grants: readReconnectGrants(readJson(required('director-log'))),
|
||||
cellRegions: cellRegionsPath ? readJson(cellRegionsPath) : {}
|
||||
})
|
||||
console.log(JSON.stringify(result, null, 2))
|
||||
}
|
||||
|
||||
if (import.meta.url === pathToFileURL(process.argv[1] ?? '').href) main()
|
||||
@@ -1,158 +0,0 @@
|
||||
import assert from 'node:assert/strict'
|
||||
import { describe, it } from 'node:test'
|
||||
import {
|
||||
measureDrainDisconnectGap,
|
||||
readDrainCloses,
|
||||
readReconnectGrants
|
||||
} from './measure-relay-drain-disconnect-gap.mjs'
|
||||
|
||||
const start = Date.parse('2026-10-06T10:00:00Z')
|
||||
const at = (seconds) => new Date(start + seconds * 1000).toISOString()
|
||||
|
||||
function close(host, seconds, reason, splices = 0) {
|
||||
return {
|
||||
timestamp: at(seconds),
|
||||
jsonPayload: {
|
||||
message:
|
||||
`[orca-relay] control closed host=${host} gen=3 state=closed ageMs=100 app="1.4.0"` +
|
||||
` splices=${splices} pending=0 code=4001 reason=${JSON.stringify(reason)}`
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function grant(host, seconds, cellId) {
|
||||
return {
|
||||
timestamp: at(seconds),
|
||||
textPayload: `[orca-relay] assignment granted lane=drain-return hinted=true host=${host} cell=${cellId}`
|
||||
}
|
||||
}
|
||||
|
||||
describe('drain disconnect gap', () => {
|
||||
it('splits moved-first, cut-off and unresolved desktops and sums dropped phone sessions', () => {
|
||||
const closes = readDrainCloses([
|
||||
close('h-moved', 30, 'migration completed', 2),
|
||||
close('h-cut', 300, 'resolve configured director', 1),
|
||||
close('h-idle', 310, 'resolve configured director'),
|
||||
close('h-lost', 320, 'resolve configured director'),
|
||||
close('h-early', -5, 'resolve configured director'),
|
||||
// A later close of the same host is its next session, not the drain.
|
||||
close('h-cut', 900, 'resolve configured director', 7)
|
||||
])
|
||||
const grants = readReconnectGrants([
|
||||
grant('h-moved', 20, 'c2'),
|
||||
grant('h-cut', 304, 'c2'),
|
||||
grant('h-idle', 330, 'c3'),
|
||||
grant('h-other', 5, 'c2'),
|
||||
grant('h-cut', 290, 'c1')
|
||||
])
|
||||
const result = measureDrainDisconnectGap({
|
||||
sourceCellId: 'c1',
|
||||
drainStartedAt: start,
|
||||
drainEndedAt: start + 1_200_000,
|
||||
controlsAtDrainStart: 5,
|
||||
closes,
|
||||
grants
|
||||
})
|
||||
assert.equal(result.closedHosts, 4)
|
||||
assert.equal(result.closeCoverage, 0.8)
|
||||
assert.equal(result.movedFirst, 1)
|
||||
assert.equal(result.cutOff, 2)
|
||||
assert.equal(result.cutOffUnresolved, 1)
|
||||
assert.equal(result.otherClose, 0)
|
||||
assert.deepEqual(result.cutOffGap, { count: 2, p50Ms: 4000, p95Ms: 20000, maxMs: 20000 })
|
||||
assert.equal(result.phoneSessionsDropped, 3)
|
||||
})
|
||||
|
||||
it('times desktops that left the source region until a grant brings them back', () => {
|
||||
const result = measureDrainDisconnectGap({
|
||||
sourceCellId: 'a1',
|
||||
drainStartedAt: start,
|
||||
drainEndedAt: start + 1_200_000,
|
||||
controlsAtDrainStart: 2,
|
||||
closes: readDrainCloses([
|
||||
close('h-away', 100, 'resolve configured director'),
|
||||
close('h-stuck', 100, 'resolve configured director')
|
||||
]),
|
||||
grants: readReconnectGrants([
|
||||
grant('h-away', 101, 'u1'),
|
||||
grant('h-away', 3701, 'a2'),
|
||||
grant('h-stuck', 102, 'u1')
|
||||
]),
|
||||
cellRegions: { a1: 'asia-east2', a2: 'asia-east2', u1: 'us-central1' }
|
||||
})
|
||||
assert.equal(result.leftSourceRegion, 2)
|
||||
assert.equal(result.leftSourceRegionStillAway, 1)
|
||||
assert.equal(result.timeUntilBack.maxMs, 3_600_000)
|
||||
})
|
||||
|
||||
it('keeps each host earliest close when the export is newest first', () => {
|
||||
const result = measureDrainDisconnectGap({
|
||||
sourceCellId: 'c1',
|
||||
drainStartedAt: start,
|
||||
drainEndedAt: start + 1_200_000,
|
||||
controlsAtDrainStart: 1,
|
||||
closes: readDrainCloses([
|
||||
close('h', 300, 'resolve configured director'),
|
||||
close('h', 60, 'resolve configured director', 3)
|
||||
]),
|
||||
grants: readReconnectGrants([grant('h', 120, 'c2')])
|
||||
})
|
||||
assert.equal(result.movedFirst, 0)
|
||||
assert.equal(result.cutOff, 1)
|
||||
assert.deepEqual(result.cutOffGap, { count: 1, p50Ms: 60000, p95Ms: 60000, maxMs: 60000 })
|
||||
assert.equal(result.phoneSessionsDropped, 3)
|
||||
})
|
||||
|
||||
it('refuses an invalid drain start time', () => {
|
||||
assert.throws(
|
||||
() =>
|
||||
measureDrainDisconnectGap({
|
||||
sourceCellId: 'c1',
|
||||
drainStartedAt: Date.parse('not a time'),
|
||||
drainEndedAt: start,
|
||||
controlsAtDrainStart: 1,
|
||||
closes: [],
|
||||
grants: []
|
||||
}),
|
||||
/drain start time is invalid/
|
||||
)
|
||||
})
|
||||
|
||||
it('leaves out closes after the drain ended and refuses a bad end time', () => {
|
||||
const input = {
|
||||
sourceCellId: 'c1',
|
||||
drainStartedAt: start,
|
||||
drainEndedAt: start + 600_000,
|
||||
controlsAtDrainStart: 1,
|
||||
closes: readDrainCloses([
|
||||
close('h-drained', 100, 'resolve configured director'),
|
||||
// The new container's session after the roll.
|
||||
close('h-new', 700, '', 4)
|
||||
]),
|
||||
grants: readReconnectGrants([grant('h-drained', 650, 'c2')])
|
||||
}
|
||||
const result = measureDrainDisconnectGap(input)
|
||||
assert.equal(result.closedHosts, 1)
|
||||
assert.equal(result.otherClose, 0)
|
||||
assert.equal(result.phoneSessionsDropped, 0)
|
||||
// A grant after the window still ends that host's gap.
|
||||
assert.equal(result.cutOffGap.maxMs, 550_000)
|
||||
for (const drainEndedAt of [Number.NaN, start]) {
|
||||
assert.throws(
|
||||
() => measureDrainDisconnectGap({ ...input, drainEndedAt }),
|
||||
/drain end time is invalid/
|
||||
)
|
||||
}
|
||||
})
|
||||
|
||||
it('ignores lines that are not the two it reads', () => {
|
||||
assert.deepEqual(readDrainCloses([{ timestamp: at(0), textPayload: 'unrelated' }]), [])
|
||||
assert.deepEqual(
|
||||
readReconnectGrants([grant('h', 0, 'c1')].map((entry) => ({
|
||||
...entry,
|
||||
textPayload: entry.textPayload.replace('hinted=true', 'hinted=false')
|
||||
}))),
|
||||
[]
|
||||
)
|
||||
})
|
||||
})
|
||||
@@ -1,509 +0,0 @@
|
||||
import { createHash } from 'node:crypto'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
import {
|
||||
addExactMigrationCells,
|
||||
applyExactAdmissionSelector,
|
||||
inspectAdmissionSelector,
|
||||
membershipWithStates,
|
||||
selectorCellState
|
||||
} from './relay-admission-selector.mjs'
|
||||
|
||||
const SHAPES = {
|
||||
staging: {
|
||||
directorOrigin: 'https://relay-staging.onorca.dev',
|
||||
domain: 'relay-staging.onorca.dev',
|
||||
allCells: ['staging-gce-c4', 'staging-gce-c5', 'staging-gce-c6', 'staging-gce-c7'],
|
||||
registrationWaves: [['staging-gce-c4'], ['staging-gce-c5', 'staging-gce-c6', 'staging-gce-c7']],
|
||||
promotionWaves: [['staging-gce-c4'], ['staging-gce-c5', 'staging-gce-c6', 'staging-gce-c7']]
|
||||
},
|
||||
production: {
|
||||
directorOrigin: 'https://relay.onorca.dev',
|
||||
domain: 'relay.onorca.dev',
|
||||
allCells: [
|
||||
'production-gce-c27', 'production-gce-c28', 'production-gce-c29', 'production-gce-c30',
|
||||
'production-gce-c31', 'production-gce-c32', 'production-gce-c33', 'production-gce-c34',
|
||||
'production-gce-c35'
|
||||
],
|
||||
// The launch set was registered together; each later cell registers alone beside it.
|
||||
registrationWaves: [
|
||||
['production-gce-c27', 'production-gce-c28', 'production-gce-c29'],
|
||||
['production-gce-c30'],
|
||||
['production-gce-c31'],
|
||||
['production-gce-c32'],
|
||||
['production-gce-c33'],
|
||||
['production-gce-c34'],
|
||||
['production-gce-c35']
|
||||
],
|
||||
promotionWaves: [
|
||||
['production-gce-c27'],
|
||||
['production-gce-c28', 'production-gce-c29'],
|
||||
['production-gce-c30'],
|
||||
['production-gce-c31'],
|
||||
['production-gce-c32'],
|
||||
['production-gce-c33'],
|
||||
['production-gce-c34'],
|
||||
['production-gce-c35']
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
const PRODUCTION_CANARY_CELL = 'production-gce-c27'
|
||||
const STAGING_PROOF_CELL = 'staging-gce-c4'
|
||||
const ASIA_REGION = 'asia-east2'
|
||||
const ASIA_CELL_SHAPE = { region: ASIA_REGION, capacityRequests: 6_000, connectionHardCap: 3_000 }
|
||||
// The US cells at the Asia shape live in the root region; every other cell here is Asia, except
|
||||
// the staging dead-cell guard cells, which keep c2's ordinary US shape (staging.tfvars).
|
||||
const STAGING_US_CELL_SHAPE = { region: 'us-central1', capacityRequests: 4_000, connectionHardCap: 600 }
|
||||
const CELL_SHAPES = {
|
||||
'production-gce-c32': { ...ASIA_CELL_SHAPE, region: 'us-central1' },
|
||||
'production-gce-c33': { ...ASIA_CELL_SHAPE, region: 'us-central1' },
|
||||
'staging-gce-c5': STAGING_US_CELL_SHAPE,
|
||||
'staging-gce-c6': STAGING_US_CELL_SHAPE,
|
||||
'staging-gce-c7': STAGING_US_CELL_SHAPE
|
||||
}
|
||||
|
||||
function cellShape(cellId) {
|
||||
return CELL_SHAPES[cellId] ?? ASIA_CELL_SHAPE
|
||||
}
|
||||
|
||||
function cellRegion(cellId) {
|
||||
return cellShape(cellId).region
|
||||
}
|
||||
|
||||
export function parseRelayAsiaAdmissionArguments(argv) {
|
||||
const values = {}
|
||||
for (let index = 0; index < argv.length; index += 2) {
|
||||
const key = argv[index]
|
||||
const value = argv[index + 1]
|
||||
if (!key?.startsWith('--') || value === undefined) throw new Error('invalid arguments')
|
||||
values[key.slice(2)] = value
|
||||
}
|
||||
for (const key of ['environment', 'mode', 'cell-ids', 'image-digest']) {
|
||||
if (!values[key]) throw new Error(`missing --${key}`)
|
||||
}
|
||||
if (!/^sha256:[a-f0-9]{64}$/.test(values['image-digest'])) {
|
||||
throw new Error('--image-digest is invalid')
|
||||
}
|
||||
if (![
|
||||
'inspect', 'initialize', 'verify', 'registered', 'register',
|
||||
'promote', 'recover-promotion', 'rollback'
|
||||
].includes(values.mode)) {
|
||||
throw new Error('--mode is invalid')
|
||||
}
|
||||
const expectedGeneration = values.mode === 'inspect'
|
||||
? undefined
|
||||
: Number(values['expected-generation'])
|
||||
if (
|
||||
values.mode !== 'inspect' &&
|
||||
(!Number.isSafeInteger(expectedGeneration) || expectedGeneration < 0)
|
||||
) {
|
||||
throw new Error('--expected-generation is invalid')
|
||||
}
|
||||
const shape = SHAPES[values.environment]
|
||||
if (!shape) throw new Error('--environment is invalid')
|
||||
const cells = values['cell-ids'].split(',').map((value) => value.trim()).filter(Boolean)
|
||||
const distinct = new Set(cells)
|
||||
if (distinct.size !== cells.length || cells.some((cell) => !shape.allCells.includes(cell))) {
|
||||
throw new Error('--cell-ids are invalid')
|
||||
}
|
||||
const exact = (expected) => JSON.stringify([...cells].sort()) === JSON.stringify([...expected].sort())
|
||||
const [launchWave] = shape.registrationWaves
|
||||
if (
|
||||
(['inspect', 'verify'].includes(values.mode) &&
|
||||
!exact(shape.allCells) && !shape.registrationWaves.some(exact)) ||
|
||||
// Generation zero predates every later cell, so the boundary freezes only the launch set.
|
||||
(values.mode === 'initialize' && !exact(launchWave)) ||
|
||||
(['register', 'registered'].includes(values.mode) && !shape.registrationWaves.some(exact)) ||
|
||||
(['promote', 'recover-promotion'].includes(values.mode) && !shape.promotionWaves.some(exact)) ||
|
||||
// Rollback takes any reviewed wave or the whole set, never a mixed partial set.
|
||||
(values.mode === 'rollback' && !exact(shape.allCells) &&
|
||||
![...shape.registrationWaves, ...shape.promotionWaves].some(exact))
|
||||
) throw new Error('--cell-ids do not match the reviewed admission wave')
|
||||
const attemptId = values['attempt-id']
|
||||
if (!['inspect', 'verify', 'registered'].includes(values.mode) &&
|
||||
!/^[A-Za-z0-9_-]{8,128}$/.test(attemptId ?? '')) {
|
||||
throw new Error('--attempt-id is invalid')
|
||||
}
|
||||
return {
|
||||
environment: values.environment,
|
||||
mode: values.mode,
|
||||
cells,
|
||||
expectedGeneration,
|
||||
expectedMembershipSha256: values['expected-membership-sha256'],
|
||||
imageDigest: values['image-digest'],
|
||||
attemptId,
|
||||
token: process.env.ORCA_RELAY_ADMIN_ID_TOKEN ?? ''
|
||||
}
|
||||
}
|
||||
|
||||
function hostname(cellId) {
|
||||
return cellId.split('-').at(-1)
|
||||
}
|
||||
|
||||
function cellOrigin(shape, cellId) {
|
||||
return `https://${hostname(cellId)}.${shape.domain}`
|
||||
}
|
||||
|
||||
async function responseJson(response, label) {
|
||||
const body = await response.json().catch(() => ({}))
|
||||
if (!response.ok) throw new Error(`${label} returned ${response.status}`)
|
||||
return body
|
||||
}
|
||||
|
||||
function defaultPost(fetchImpl, token) {
|
||||
return async (url, body) => await responseJson(await fetchImpl(url, {
|
||||
method: 'POST',
|
||||
headers: { authorization: `Bearer ${token}`, 'content-type': 'application/json' },
|
||||
body: JSON.stringify(body),
|
||||
signal: AbortSignal.timeout(30_000)
|
||||
}), new URL(url).pathname)
|
||||
}
|
||||
|
||||
async function verifyRuntime(fetchImpl, post, shape, cellId, imageDigest, requireDirector) {
|
||||
const origin = cellOrigin(shape, cellId)
|
||||
const [health, ready, runtime] = await Promise.all([
|
||||
fetchImpl(`${origin}/health`, { redirect: 'error', signal: AbortSignal.timeout(8_000) }),
|
||||
fetchImpl(`${origin}/ready`, { redirect: 'error', signal: AbortSignal.timeout(8_000) }),
|
||||
post(`${origin}/v1/admin/runtime-status`, { v: 1 })
|
||||
])
|
||||
if (!health.ok || !ready.ok) throw new Error(`${cellId} is not ready`)
|
||||
if (
|
||||
runtime.cellId !== cellId ||
|
||||
runtime.cellUrl !== origin ||
|
||||
runtime.region !== cellRegion(cellId) ||
|
||||
runtime.imageDigest !== imageDigest ||
|
||||
runtime.draining !== false ||
|
||||
runtime.connectionCapacity?.hardCap !== cellShape(cellId).connectionHardCap ||
|
||||
runtime.connectionCapacity?.unobservedBound !== 60
|
||||
) throw new Error(`${cellId} runtime does not match its reviewed shape`)
|
||||
if (requireDirector) {
|
||||
const result = await post(`${shape.directorOrigin}/v1/admin/cell-status`, { v: 1, cellId })
|
||||
if (
|
||||
result.status?.cellUrl !== origin ||
|
||||
result.status?.runtime?.heartbeatFresh !== true ||
|
||||
result.status?.runtime?.ready !== true
|
||||
) throw new Error(`${cellId} has no fresh ready director heartbeat`)
|
||||
}
|
||||
}
|
||||
|
||||
function membershipStates(selector, cells) {
|
||||
return Object.fromEntries(cells.map((cellId) => [cellId, selectorCellState(selector, cellId)]))
|
||||
}
|
||||
|
||||
function inspectedMembershipStates(selector, cells) {
|
||||
const known = new Set([
|
||||
...selector.membership.existingOnly,
|
||||
...selector.membership.migrationOnly,
|
||||
...selector.membership.general
|
||||
])
|
||||
return Object.fromEntries(cells.map((cellId) => [
|
||||
cellId,
|
||||
known.has(cellId) ? selectorCellState(selector, cellId) : 'absent'
|
||||
]))
|
||||
}
|
||||
|
||||
function sameMembership(left, right) {
|
||||
return JSON.stringify(left) === JSON.stringify(right)
|
||||
}
|
||||
|
||||
function membershipSha256(membership) {
|
||||
return createHash('sha256').update(JSON.stringify(membership)).digest('hex')
|
||||
}
|
||||
|
||||
async function initializeAdmissionBoundary(post, selectorPost, shape, config, current) {
|
||||
if (config.expectedGeneration !== 0) {
|
||||
throw new Error('admission boundary initialization requires generation 0')
|
||||
}
|
||||
if (
|
||||
!/^[a-f0-9]{64}$/.test(config.expectedMembershipSha256 ?? '') ||
|
||||
membershipSha256(current.selector.membership) !== config.expectedMembershipSha256
|
||||
) {
|
||||
throw new Error('admission membership changed before boundary initialization')
|
||||
}
|
||||
const targetStates = inspectedMembershipStates(current.selector, config.cells)
|
||||
if (Object.values(targetStates).some((state) => state !== 'absent')) {
|
||||
throw new Error('Asia cell exists before admission boundary initialization')
|
||||
}
|
||||
const intendedMembership = current.intent?.previousMembership ?? current.selector.membership
|
||||
const exactCommitted = (inspection) =>
|
||||
inspection.intent?.state === 'committed' &&
|
||||
inspection.intent.expectedGeneration === 0 &&
|
||||
inspection.selector.generation === 1 &&
|
||||
inspection.selector.attemptId === config.attemptId &&
|
||||
sameMembership(inspection.intent.previousMembership, intendedMembership) &&
|
||||
sameMembership(inspection.intent.membership, intendedMembership) &&
|
||||
sameMembership(inspection.selector.membership, intendedMembership)
|
||||
const exactUnchanged = (inspection) =>
|
||||
inspection.intent?.state === 'unchanged' &&
|
||||
inspection.intent.expectedGeneration === 0 &&
|
||||
inspection.selector.generation === 0 &&
|
||||
sameMembership(inspection.intent.previousMembership, intendedMembership) &&
|
||||
sameMembership(inspection.intent.membership, intendedMembership) &&
|
||||
sameMembership(inspection.selector.membership, intendedMembership)
|
||||
if (exactCommitted(current)) {
|
||||
return {
|
||||
mode: config.mode,
|
||||
generation: current.selector.generation,
|
||||
states: inspectedMembershipStates(current.selector, config.cells),
|
||||
recovered: true
|
||||
}
|
||||
}
|
||||
if (current.intent && !exactUnchanged(current)) {
|
||||
throw new Error('admission boundary initialization attempt diverged')
|
||||
}
|
||||
const request = {
|
||||
v: 1,
|
||||
attemptId: config.attemptId,
|
||||
expectedGeneration: 0,
|
||||
expectedMembershipSha256: config.expectedMembershipSha256,
|
||||
membership: intendedMembership
|
||||
}
|
||||
let applyError
|
||||
for (let attempt = 0; attempt < 2; attempt++) {
|
||||
try {
|
||||
await post(`${shape.directorOrigin}/v1/admin/admission-selector/apply`, request)
|
||||
} catch (error) {
|
||||
applyError = error
|
||||
}
|
||||
const verified = await inspectAdmissionSelector(selectorPost, config.attemptId)
|
||||
if (exactCommitted(verified)) {
|
||||
return {
|
||||
mode: config.mode,
|
||||
generation: verified.selector.generation,
|
||||
states: targetStates,
|
||||
recovered: current.intent !== null || applyError !== undefined || attempt > 0
|
||||
}
|
||||
}
|
||||
if (!exactUnchanged(verified)) {
|
||||
throw new Error('admission boundary initialization did not commit exactly', {
|
||||
cause: applyError
|
||||
})
|
||||
}
|
||||
}
|
||||
throw new Error('admission boundary initialization remained unchanged after retry', {
|
||||
cause: applyError
|
||||
})
|
||||
}
|
||||
|
||||
export async function operateRelayAsiaAdmission(config, dependencies = {}) {
|
||||
const shape = SHAPES[config.environment]
|
||||
const fetchImpl = dependencies.fetch ?? fetch
|
||||
const post = dependencies.post ?? defaultPost(fetchImpl, config.token)
|
||||
// Only the C4 proof keeps the server-enforced C4-only route (the Asia proof identity can call
|
||||
// nothing else); other staging cells go through the general route, as production does.
|
||||
const stagingProof = config.environment === 'staging' &&
|
||||
config.cells.length === 1 && config.cells[0] === STAGING_PROOF_CELL
|
||||
const selectorPost = (path, body) => {
|
||||
if (!stagingProof || path !== '/v1/admin/admission-selector/apply') {
|
||||
return post(`${shape.directorOrigin}${path}`, body)
|
||||
}
|
||||
const state = selectorCellState({ membership: body.membership }, STAGING_PROOF_CELL)
|
||||
if (!['general', 'migration-only'].includes(state)) {
|
||||
throw new Error('staging proof can only transition C4 between reviewed states')
|
||||
}
|
||||
return post(`${shape.directorOrigin}/v1/admin/admission-selector/apply-staging-asia-proof`, {
|
||||
v: 1,
|
||||
attemptId: body.attemptId,
|
||||
expectedGeneration: body.expectedGeneration,
|
||||
state
|
||||
})
|
||||
}
|
||||
const current = await inspectAdmissionSelector(
|
||||
selectorPost,
|
||||
['inspect', 'verify', 'registered'].includes(config.mode) ? undefined : config.attemptId
|
||||
)
|
||||
if (config.mode === 'inspect') {
|
||||
return {
|
||||
mode: config.mode,
|
||||
generation: current.selector.generation,
|
||||
membership: current.selector.membership,
|
||||
membershipSha256: membershipSha256(current.selector.membership),
|
||||
states: inspectedMembershipStates(current.selector, config.cells)
|
||||
}
|
||||
}
|
||||
if (
|
||||
!current.intent &&
|
||||
current.selector.generation !== config.expectedGeneration
|
||||
) {
|
||||
throw new Error('admission selector generation changed')
|
||||
}
|
||||
if (current.intent && current.intent.expectedGeneration !== config.expectedGeneration) {
|
||||
throw new Error('admission attempt generation does not match')
|
||||
}
|
||||
if (config.mode === 'initialize') {
|
||||
return await initializeAdmissionBoundary(post, selectorPost, shape, config, current)
|
||||
}
|
||||
if (config.mode === 'recover-promotion') {
|
||||
if (config.cells.every(
|
||||
(cellId) => selectorCellState(current.selector, cellId) === 'migration-only'
|
||||
)) {
|
||||
return {
|
||||
mode: config.mode,
|
||||
promoted: false,
|
||||
generation: current.selector.generation,
|
||||
states: membershipStates(current.selector, config.cells)
|
||||
}
|
||||
}
|
||||
if (!current.intent) {
|
||||
if (
|
||||
current.selector.generation !== config.expectedGeneration
|
||||
) throw new Error('promotion state changed without the reviewed attempt')
|
||||
return {
|
||||
mode: config.mode,
|
||||
promoted: false,
|
||||
generation: current.selector.generation,
|
||||
states: membershipStates(current.selector, config.cells)
|
||||
}
|
||||
}
|
||||
const expectedMembership = membershipWithStates(
|
||||
{ membership: current.intent.previousMembership },
|
||||
Object.fromEntries(config.cells.map((cellId) => [cellId, 'general']))
|
||||
)
|
||||
if (
|
||||
current.intent.state !== 'committed' ||
|
||||
JSON.stringify(current.intent.membership) !== JSON.stringify(expectedMembership) ||
|
||||
config.cells.some((cellId) => selectorCellState(current.selector, cellId) !== 'general')
|
||||
) throw new Error('promotion attempt is not the current general state')
|
||||
return {
|
||||
mode: config.mode,
|
||||
promoted: true,
|
||||
generation: current.selector.generation,
|
||||
states: membershipStates(current.selector, config.cells)
|
||||
}
|
||||
}
|
||||
if (config.mode === 'rollback') {
|
||||
for (const cellId of config.cells) {
|
||||
if (!['general', 'migration-only'].includes(selectorCellState(current.selector, cellId))) {
|
||||
throw new Error(`${cellId} cannot roll back to migration-only`)
|
||||
}
|
||||
}
|
||||
} else if (config.mode === 'register') {
|
||||
const known = new Set([
|
||||
...current.selector.membership.existingOnly,
|
||||
...current.selector.membership.migrationOnly,
|
||||
...current.selector.membership.general
|
||||
])
|
||||
if (!current.intent && config.cells.some((cellId) => known.has(cellId))) {
|
||||
throw new Error('Asia cell is already registered')
|
||||
}
|
||||
await Promise.all(config.cells.map((cellId) =>
|
||||
verifyRuntime(fetchImpl, post, shape, cellId, config.imageDigest, false)
|
||||
))
|
||||
} else if (config.mode !== 'registered') {
|
||||
await Promise.all(config.cells.map((cellId) =>
|
||||
verifyRuntime(fetchImpl, post, shape, cellId, config.imageDigest, true)
|
||||
))
|
||||
}
|
||||
if (config.mode === 'registered') {
|
||||
if (config.cells.some(
|
||||
(cellId) => selectorCellState(current.selector, cellId) !== 'migration-only'
|
||||
)) throw new Error('Asia cells are not registered migration-only')
|
||||
await Promise.all(config.cells.map((cellId) =>
|
||||
verifyRuntime(fetchImpl, post, shape, cellId, config.imageDigest, false)
|
||||
))
|
||||
}
|
||||
if (['verify', 'registered'].includes(config.mode)) {
|
||||
return {
|
||||
mode: config.mode,
|
||||
generation: current.selector.generation,
|
||||
states: membershipStates(current.selector, config.cells)
|
||||
}
|
||||
}
|
||||
if (config.mode === 'register') {
|
||||
if (current.intent) {
|
||||
const expectedCells = new Set(config.cells)
|
||||
const addedCells = current.intent.membership.migrationOnly.filter(
|
||||
(cellId) => !current.intent.previousMembership.migrationOnly.includes(cellId)
|
||||
)
|
||||
if (
|
||||
current.intent.state !== 'committed' ||
|
||||
addedCells.length !== expectedCells.size ||
|
||||
addedCells.some((cellId) => !expectedCells.has(cellId)) ||
|
||||
current.selector.generation !== config.expectedGeneration + 1 ||
|
||||
JSON.stringify(current.selector.membership) !== JSON.stringify(current.intent.membership)
|
||||
) {
|
||||
throw new Error('admission attempt does not match the requested Asia registration')
|
||||
}
|
||||
return {
|
||||
mode: config.mode,
|
||||
generation: current.selector.generation,
|
||||
states: membershipStates(current.selector, config.cells),
|
||||
recovered: true
|
||||
}
|
||||
}
|
||||
const result = await addExactMigrationCells(
|
||||
selectorPost,
|
||||
{
|
||||
attemptId: config.attemptId,
|
||||
cells: config.cells.map((cellId) => ({
|
||||
cellId,
|
||||
cellUrl: cellOrigin(shape, cellId),
|
||||
region: cellRegion(cellId),
|
||||
capacityRequests: cellShape(cellId).capacityRequests,
|
||||
connectionHardCap: cellShape(cellId).connectionHardCap,
|
||||
connectionUnobservedBound: 60
|
||||
}))
|
||||
},
|
||||
{ expectedCurrentSelector: current.selector }
|
||||
)
|
||||
return { mode: config.mode, generation: result.selector.generation, states: membershipStates(result.selector, config.cells) }
|
||||
}
|
||||
const desiredState = config.mode === 'promote' ? 'general' : 'migration-only'
|
||||
if (current.intent) {
|
||||
const expectedMembership = membershipWithStates(
|
||||
{ membership: current.intent.previousMembership },
|
||||
Object.fromEntries(config.cells.map((cellId) => [cellId, desiredState]))
|
||||
)
|
||||
if (
|
||||
current.intent.state !== 'committed' ||
|
||||
JSON.stringify(current.intent.membership) !== JSON.stringify(expectedMembership) ||
|
||||
current.selector.generation !== config.expectedGeneration + 1 ||
|
||||
JSON.stringify(current.selector.membership) !== JSON.stringify(current.intent.membership)
|
||||
) {
|
||||
throw new Error('admission attempt does not match the requested Asia transition')
|
||||
}
|
||||
return {
|
||||
mode: config.mode,
|
||||
generation: current.selector.generation,
|
||||
states: membershipStates(current.selector, config.cells),
|
||||
recovered: true
|
||||
}
|
||||
}
|
||||
if (config.mode === 'promote' && config.cells.some(
|
||||
(cellId) => selectorCellState(current.selector, cellId) !== 'migration-only'
|
||||
)) throw new Error('Asia promotion requires migration-only cells')
|
||||
// The Asia launch order binds Asia cells only; a US cell is proven by its own canary.
|
||||
const asiaWave = config.cells.every((cellId) => cellRegion(cellId) === ASIA_REGION)
|
||||
if (
|
||||
config.mode === 'promote' &&
|
||||
asiaWave &&
|
||||
config.environment === 'production' &&
|
||||
!config.cells.includes(PRODUCTION_CANARY_CELL) &&
|
||||
selectorCellState(current.selector, PRODUCTION_CANARY_CELL) !== 'general'
|
||||
) {
|
||||
throw new Error('Asia expansion requires the C27 canary to be general')
|
||||
}
|
||||
const [launchWave] = shape.registrationWaves
|
||||
if (
|
||||
config.mode === 'promote' &&
|
||||
asiaWave &&
|
||||
!config.cells.some((cellId) => launchWave.includes(cellId)) &&
|
||||
launchWave.some((cellId) => selectorCellState(current.selector, cellId) !== 'general')
|
||||
) {
|
||||
throw new Error('a later Asia cell requires every launch cell to be general')
|
||||
}
|
||||
const result = await applyExactAdmissionSelector(
|
||||
selectorPost,
|
||||
membershipWithStates(current.selector, Object.fromEntries(
|
||||
config.cells.map((cellId) => [cellId, desiredState])
|
||||
)),
|
||||
{ attemptId: config.attemptId, expectedCurrentSelector: current.selector }
|
||||
)
|
||||
return { mode: config.mode, generation: result.selector.generation, states: membershipStates(result.selector, config.cells) }
|
||||
}
|
||||
|
||||
if (process.argv[1] === fileURLToPath(import.meta.url)) {
|
||||
const config = parseRelayAsiaAdmissionArguments(process.argv.slice(2))
|
||||
if (!config.token) throw new Error('ORCA_RELAY_ADMIN_ID_TOKEN is required')
|
||||
console.log(JSON.stringify(await operateRelayAsiaAdmission(config)))
|
||||
}
|
||||
@@ -1,967 +0,0 @@
|
||||
import assert from 'node:assert/strict'
|
||||
import { spawnSync } from 'node:child_process'
|
||||
import { createHash } from 'node:crypto'
|
||||
import { mkdtempSync, readFileSync, rmSync } from 'node:fs'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { join } from 'node:path'
|
||||
import { test } from 'node:test'
|
||||
import {
|
||||
operateRelayAsiaAdmission,
|
||||
parseRelayAsiaAdmissionArguments
|
||||
} from './operate-relay-asia-admission.mjs'
|
||||
import { readRelayWorkflow } from './relay-repository.mjs'
|
||||
|
||||
const digest = `sha256:${'a'.repeat(64)}`
|
||||
const membershipDigest = (membership) =>
|
||||
createHash('sha256').update(JSON.stringify(membership)).digest('hex')
|
||||
|
||||
function harness(initialSelector, runtimeDigests = {}, runtimeRegions = {}, runtimeHardCaps = {}) {
|
||||
const initialMembership = structuredClone(initialSelector.membership)
|
||||
let selector = structuredClone(initialSelector)
|
||||
const intents = new Map()
|
||||
const requests = []
|
||||
let fetches = 0
|
||||
let failAfterIntent = false
|
||||
const post = async (url, body) => {
|
||||
const parsed = new URL(url)
|
||||
requests.push({ path: parsed.pathname, body })
|
||||
if (parsed.pathname === '/v1/admin/runtime-status') {
|
||||
const [cell, domain] = parsed.hostname.split(/\.(.*)/)
|
||||
const cellId = `${domain.startsWith('relay-staging.') ? 'staging' : 'production'}-gce-${cell}`
|
||||
return {
|
||||
cellId,
|
||||
cellUrl: parsed.origin,
|
||||
region: runtimeRegions[cellId] ?? 'asia-east2',
|
||||
imageDigest: runtimeDigests[cellId] ?? digest,
|
||||
draining: false,
|
||||
connectionCapacity: { hardCap: runtimeHardCaps[cellId] ?? 3_000, unobservedBound: 60 }
|
||||
}
|
||||
}
|
||||
if (parsed.pathname === '/v1/admin/cell-status') {
|
||||
return {
|
||||
status: {
|
||||
cellUrl: `https://${body.cellId.split('-').at(-1)}.${parsed.hostname}`,
|
||||
runtime: { heartbeatFresh: true, ready: true }
|
||||
}
|
||||
}
|
||||
}
|
||||
if (parsed.pathname.endsWith('/status')) {
|
||||
return { selector, intent: body.attemptId ? intents.get(body.attemptId) ?? null : null }
|
||||
}
|
||||
if (parsed.pathname.endsWith('/add-migration-cells')) {
|
||||
selector = {
|
||||
generation: selector.generation + 1,
|
||||
attemptId: body.attemptId,
|
||||
membership: {
|
||||
...selector.membership,
|
||||
migrationOnly: [...selector.membership.migrationOnly, ...body.cells.map((cell) => cell.cellId)].sort()
|
||||
}
|
||||
}
|
||||
} else if (parsed.pathname.endsWith('/apply-staging-asia-proof')) {
|
||||
const membership = structuredClone(selector.membership)
|
||||
membership.migrationOnly = membership.migrationOnly.filter((cell) => cell !== 'staging-gce-c4')
|
||||
membership.general = membership.general.filter((cell) => cell !== 'staging-gce-c4')
|
||||
membership[body.state === 'general' ? 'general' : 'migrationOnly'].push('staging-gce-c4')
|
||||
selector = { generation: selector.generation + 1, attemptId: body.attemptId, membership }
|
||||
} else if (parsed.pathname.endsWith('/apply')) {
|
||||
if (
|
||||
body.expectedMembershipSha256 &&
|
||||
body.expectedMembershipSha256 !== membershipDigest(selector.membership)
|
||||
) throw new Error('admission_selector_membership_mismatch')
|
||||
if (failAfterIntent) {
|
||||
failAfterIntent = false
|
||||
intents.set(body.attemptId, {
|
||||
state: 'unchanged',
|
||||
expectedGeneration: body.expectedGeneration,
|
||||
previousMembership: structuredClone(selector.membership),
|
||||
membership: structuredClone(body.membership)
|
||||
})
|
||||
throw new Error('failure after intent persistence')
|
||||
}
|
||||
selector = { generation: selector.generation + 1, attemptId: body.attemptId, membership: body.membership }
|
||||
} else throw new Error(`unexpected ${parsed.pathname}`)
|
||||
intents.set(body.attemptId, {
|
||||
state: 'committed', expectedGeneration: body.expectedGeneration,
|
||||
previousMembership: initialSelector.membership,
|
||||
membership: selector.membership
|
||||
})
|
||||
return { changed: true, selector }
|
||||
}
|
||||
const fetch = async () => {
|
||||
fetches++
|
||||
return new Response(null, { status: 200 })
|
||||
}
|
||||
const commitWithoutResponse = async (path, body) => {
|
||||
await post(`https://relay.onorca.dev${path}`, body)
|
||||
throw new Error('response lost after commit')
|
||||
}
|
||||
return {
|
||||
post, fetch, requests, commitWithoutResponse,
|
||||
failNextApplyAfterIntent: () => (failAfterIntent = true),
|
||||
apply: async (attemptId, membership) => await post(
|
||||
'https://relay.onorca.dev/v1/admin/admission-selector/apply',
|
||||
{ attemptId, expectedGeneration: selector.generation, membership }
|
||||
),
|
||||
fetchCount: () => fetches, selector: () => selector
|
||||
}
|
||||
}
|
||||
|
||||
const baseSelector = {
|
||||
generation: 7,
|
||||
membership: { existingOnly: [], migrationOnly: [], general: ['production-gce-c26'] }
|
||||
}
|
||||
|
||||
test('inspects generation zero without requiring target registration or making a mutation', async () => {
|
||||
const subject = harness({
|
||||
generation: 0,
|
||||
membership: {
|
||||
existingOnly: ['staging-gce-c3'],
|
||||
migrationOnly: [],
|
||||
general: ['staging-gce-c1', 'staging-gce-c2']
|
||||
}
|
||||
})
|
||||
const result = await operateRelayAsiaAdmission({
|
||||
environment: 'staging', mode: 'inspect', cells: ['staging-gce-c4'],
|
||||
imageDigest: digest, token: 'not-logged'
|
||||
}, subject)
|
||||
assert.equal(result.generation, 0)
|
||||
assert.equal(result.states['staging-gce-c4'], 'absent')
|
||||
assert.deepEqual(result.membership, subject.selector().membership)
|
||||
assert.equal(result.membershipSha256, membershipDigest(subject.selector().membership))
|
||||
assert.deepEqual(subject.requests.map(({ path }) => path), [
|
||||
'/v1/admin/admission-selector/status'
|
||||
])
|
||||
})
|
||||
|
||||
test('initializes generation zero without changing membership', async () => {
|
||||
const membership = {
|
||||
existingOnly: ['staging-gce-c3'],
|
||||
migrationOnly: [],
|
||||
general: ['staging-gce-c1', 'staging-gce-c2']
|
||||
}
|
||||
const subject = harness({ generation: 0, membership })
|
||||
const result = await operateRelayAsiaAdmission({
|
||||
environment: 'staging', mode: 'initialize', cells: ['staging-gce-c4'],
|
||||
expectedGeneration: 0, imageDigest: digest,
|
||||
expectedMembershipSha256: membershipDigest(membership),
|
||||
attemptId: 'asia_boundary_0', token: 'not-logged'
|
||||
}, subject)
|
||||
const request = subject.requests.find(({ path }) => path.endsWith('/apply'))
|
||||
assert.deepEqual(request.body, {
|
||||
v: 1,
|
||||
attemptId: 'asia_boundary_0',
|
||||
expectedGeneration: 0,
|
||||
expectedMembershipSha256: membershipDigest(membership),
|
||||
membership
|
||||
})
|
||||
assert.equal(result.generation, 1)
|
||||
assert.equal(result.states['staging-gce-c4'], 'absent')
|
||||
assert.deepEqual(subject.selector().membership, membership)
|
||||
})
|
||||
|
||||
test('retries the same fingerprint-bound initialization after intent persistence', async () => {
|
||||
const membership = {
|
||||
existingOnly: ['staging-gce-c3'],
|
||||
migrationOnly: [],
|
||||
general: ['staging-gce-c1', 'staging-gce-c2']
|
||||
}
|
||||
const subject = harness({ generation: 0, membership })
|
||||
subject.failNextApplyAfterIntent()
|
||||
const result = await operateRelayAsiaAdmission({
|
||||
environment: 'staging', mode: 'initialize', cells: ['staging-gce-c4'],
|
||||
expectedGeneration: 0, imageDigest: digest,
|
||||
expectedMembershipSha256: membershipDigest(membership),
|
||||
attemptId: 'asia_boundary_intent_retry', token: 'not-logged'
|
||||
}, subject)
|
||||
const applies = subject.requests.filter(({ path }) => path.endsWith('/apply'))
|
||||
assert.equal(applies.length, 2)
|
||||
assert.deepEqual(applies[1].body, applies[0].body)
|
||||
assert.equal(result.recovered, true)
|
||||
assert.equal(result.generation, 1)
|
||||
assert.deepEqual(subject.selector().membership, membership)
|
||||
})
|
||||
|
||||
test('recovers a committed generation-zero initialization', async () => {
|
||||
const membership = {
|
||||
existingOnly: ['staging-gce-c3'],
|
||||
migrationOnly: [],
|
||||
general: ['staging-gce-c1', 'staging-gce-c2']
|
||||
}
|
||||
const subject = harness({ generation: 0, membership })
|
||||
const config = {
|
||||
environment: 'staging', mode: 'initialize', cells: ['staging-gce-c4'],
|
||||
expectedGeneration: 0, imageDigest: digest,
|
||||
expectedMembershipSha256: membershipDigest(membership),
|
||||
attemptId: 'asia_boundary_retry', token: 'not-logged'
|
||||
}
|
||||
await assert.rejects(subject.commitWithoutResponse(
|
||||
'/v1/admin/admission-selector/apply',
|
||||
{
|
||||
v: 1,
|
||||
attemptId: config.attemptId,
|
||||
expectedGeneration: 0,
|
||||
expectedMembershipSha256: membershipDigest(membership),
|
||||
membership
|
||||
}
|
||||
), /response lost after commit/)
|
||||
const recovered = await operateRelayAsiaAdmission(config, subject)
|
||||
assert.equal(recovered.recovered, true)
|
||||
assert.equal(recovered.generation, 1)
|
||||
assert.deepEqual(subject.selector().membership, membership)
|
||||
})
|
||||
|
||||
test('rejects generation-zero membership drift after inspect', async () => {
|
||||
const inspected = {
|
||||
existingOnly: ['staging-gce-c3'],
|
||||
migrationOnly: [],
|
||||
general: ['staging-gce-c1', 'staging-gce-c2']
|
||||
}
|
||||
const changed = {
|
||||
existingOnly: ['staging-gce-c2', 'staging-gce-c3'],
|
||||
migrationOnly: [],
|
||||
general: ['staging-gce-c1']
|
||||
}
|
||||
const subject = harness({ generation: 0, membership: changed })
|
||||
await assert.rejects(operateRelayAsiaAdmission({
|
||||
environment: 'staging', mode: 'initialize', cells: ['staging-gce-c4'],
|
||||
expectedGeneration: 0, imageDigest: digest,
|
||||
expectedMembershipSha256: membershipDigest(inspected),
|
||||
attemptId: 'asia_boundary_drift', token: 'not-logged'
|
||||
}, subject), /membership changed/)
|
||||
assert.equal(subject.requests.some(({ path }) => path.endsWith('/apply')), false)
|
||||
})
|
||||
|
||||
test('registers all three Asia cells atomically with region and exact limits', async () => {
|
||||
const subject = harness(baseSelector)
|
||||
const result = await operateRelayAsiaAdmission({
|
||||
environment: 'production', mode: 'register',
|
||||
cells: ['production-gce-c27', 'production-gce-c28', 'production-gce-c29'],
|
||||
expectedGeneration: 7, imageDigest: digest, attemptId: 'asia_register_7', token: 'not-logged'
|
||||
}, subject)
|
||||
const request = subject.requests.find(({ path }) => path.endsWith('/add-migration-cells'))
|
||||
assert.equal(request.body.cells.length, 3)
|
||||
assert.ok(request.body.cells.every((cell) =>
|
||||
cell.region === 'asia-east2' && cell.capacityRequests === 6_000 &&
|
||||
cell.connectionHardCap === 3_000 && cell.connectionUnobservedBound === 60
|
||||
))
|
||||
assert.equal(result.generation, 8)
|
||||
assert.deepEqual(new Set(Object.values(result.states)), new Set(['migration-only']))
|
||||
})
|
||||
|
||||
test('promotes the canary only after runtime and director-heartbeat checks', async () => {
|
||||
const subject = harness({
|
||||
generation: 8,
|
||||
membership: { existingOnly: [], migrationOnly: ['production-gce-c27'], general: ['production-gce-c26'] }
|
||||
})
|
||||
const result = await operateRelayAsiaAdmission({
|
||||
environment: 'production', mode: 'promote', cells: ['production-gce-c27'],
|
||||
expectedGeneration: 8, imageDigest: digest, attemptId: 'asia_promote_8', token: 'not-logged'
|
||||
}, subject)
|
||||
assert.equal(subject.fetchCount(), 2)
|
||||
assert.equal(result.states['production-gce-c27'], 'general')
|
||||
})
|
||||
|
||||
test('checks registered migration-only cells before director configuration without requiring heartbeat', async () => {
|
||||
const subject = harness({
|
||||
generation: 8,
|
||||
membership: {
|
||||
existingOnly: [],
|
||||
migrationOnly: ['production-gce-c27', 'production-gce-c28', 'production-gce-c29'],
|
||||
general: ['production-gce-c26']
|
||||
}
|
||||
})
|
||||
const result = await operateRelayAsiaAdmission({
|
||||
environment: 'production', mode: 'registered',
|
||||
cells: ['production-gce-c27', 'production-gce-c28', 'production-gce-c29'],
|
||||
expectedGeneration: 8, imageDigest: digest, token: 'not-logged'
|
||||
}, subject)
|
||||
assert.equal(subject.fetchCount(), 6)
|
||||
assert.equal(subject.requests.filter(({ path }) => path === '/v1/admin/cell-status').length, 0)
|
||||
assert.deepEqual(new Set(Object.values(result.states)), new Set(['migration-only']))
|
||||
})
|
||||
|
||||
test('rolls back admission without requiring an unhealthy runtime to answer', async () => {
|
||||
const subject = harness({
|
||||
generation: 9,
|
||||
membership: { existingOnly: [], migrationOnly: [], general: ['production-gce-c26', 'production-gce-c27'] }
|
||||
})
|
||||
const result = await operateRelayAsiaAdmission({
|
||||
environment: 'production', mode: 'rollback', cells: ['production-gce-c27'],
|
||||
expectedGeneration: 9, imageDigest: digest, attemptId: 'asia_rollback_9', token: 'not-logged'
|
||||
}, subject)
|
||||
assert.equal(subject.fetchCount(), 0)
|
||||
assert.equal(result.states['production-gce-c27'], 'migration-only')
|
||||
})
|
||||
|
||||
test('uses the server-enforced C4-only route for staging proof transitions', async () => {
|
||||
const subject = harness({
|
||||
generation: 3,
|
||||
membership: {
|
||||
existingOnly: ['staging-gce-c1'],
|
||||
migrationOnly: [],
|
||||
general: ['staging-gce-c2', 'staging-gce-c4']
|
||||
}
|
||||
})
|
||||
const result = await operateRelayAsiaAdmission({
|
||||
environment: 'staging', mode: 'rollback', cells: ['staging-gce-c4'],
|
||||
expectedGeneration: 3, imageDigest: digest, attemptId: 'asia_staging_rollback',
|
||||
token: 'not-logged'
|
||||
}, subject)
|
||||
const request = subject.requests.find(
|
||||
({ path }) => path.endsWith('/apply-staging-asia-proof')
|
||||
)
|
||||
assert.deepEqual(request.body, {
|
||||
v: 1,
|
||||
attemptId: 'asia_staging_rollback',
|
||||
expectedGeneration: 3,
|
||||
state: 'migration-only'
|
||||
})
|
||||
assert.deepEqual(subject.selector().membership, {
|
||||
existingOnly: ['staging-gce-c1'],
|
||||
migrationOnly: ['staging-gce-c4'],
|
||||
general: ['staging-gce-c2']
|
||||
})
|
||||
assert.equal(result.states['staging-gce-c4'], 'migration-only')
|
||||
})
|
||||
|
||||
test('fails closed when the exact selector generation moved', async () => {
|
||||
const subject = harness(baseSelector)
|
||||
await assert.rejects(operateRelayAsiaAdmission({
|
||||
environment: 'production', mode: 'register',
|
||||
cells: ['production-gce-c27', 'production-gce-c28', 'production-gce-c29'],
|
||||
expectedGeneration: 6, imageDigest: digest, attemptId: 'asia_register_6', token: 'not-logged'
|
||||
}, subject), /generation changed/)
|
||||
assert.equal(subject.fetchCount(), 0)
|
||||
})
|
||||
|
||||
test('recovers a committed registration when the workflow retries the original generation', async () => {
|
||||
const subject = harness(baseSelector)
|
||||
const config = {
|
||||
environment: 'production', mode: 'register',
|
||||
cells: ['production-gce-c27', 'production-gce-c28', 'production-gce-c29'],
|
||||
expectedGeneration: 7, imageDigest: digest, attemptId: 'asia_register_retry',
|
||||
token: 'not-logged'
|
||||
}
|
||||
await assert.rejects(subject.commitWithoutResponse(
|
||||
'/v1/admin/admission-selector/add-migration-cells',
|
||||
{
|
||||
v: 1,
|
||||
attemptId: config.attemptId,
|
||||
expectedGeneration: config.expectedGeneration,
|
||||
cells: config.cells.map((cellId) => ({ cellId }))
|
||||
}
|
||||
), /response lost after commit/)
|
||||
const recovered = await operateRelayAsiaAdmission(config, subject)
|
||||
assert.equal(recovered.recovered, true)
|
||||
assert.equal(recovered.generation, 8)
|
||||
})
|
||||
|
||||
test('recovers a committed promotion when the workflow retries the original generation', async () => {
|
||||
const subject = harness({
|
||||
generation: 8,
|
||||
membership: {
|
||||
existingOnly: [], migrationOnly: ['production-gce-c27'], general: ['production-gce-c26']
|
||||
}
|
||||
})
|
||||
const config = {
|
||||
environment: 'production', mode: 'promote', cells: ['production-gce-c27'],
|
||||
expectedGeneration: 8, imageDigest: digest, attemptId: 'asia_promote_retry',
|
||||
token: 'not-logged'
|
||||
}
|
||||
await assert.rejects(subject.commitWithoutResponse(
|
||||
'/v1/admin/admission-selector/apply',
|
||||
{
|
||||
v: 1,
|
||||
attemptId: config.attemptId,
|
||||
expectedGeneration: config.expectedGeneration,
|
||||
membership: {
|
||||
existingOnly: [], migrationOnly: [],
|
||||
general: ['production-gce-c26', 'production-gce-c27']
|
||||
}
|
||||
}
|
||||
), /response lost after commit/)
|
||||
const recovered = await operateRelayAsiaAdmission(config, subject)
|
||||
assert.equal(recovered.recovered, true)
|
||||
assert.equal(recovered.generation, 9)
|
||||
assert.equal(recovered.states['production-gce-c27'], 'general')
|
||||
})
|
||||
|
||||
test('inspects an ambiguous promotion without creating a new transition', async () => {
|
||||
const untouched = harness({
|
||||
generation: 8,
|
||||
membership: {
|
||||
existingOnly: [], migrationOnly: ['production-gce-c27'], general: ['production-gce-c26']
|
||||
}
|
||||
})
|
||||
const config = {
|
||||
environment: 'production', mode: 'recover-promotion', cells: ['production-gce-c27'],
|
||||
expectedGeneration: 8, imageDigest: digest, attemptId: 'asia_recover_promote',
|
||||
token: 'not-logged'
|
||||
}
|
||||
const absent = await operateRelayAsiaAdmission(config, untouched)
|
||||
assert.equal(absent.promoted, false)
|
||||
assert.equal(untouched.requests.some(({ path }) => path.endsWith('/apply')), false)
|
||||
|
||||
const committed = harness({
|
||||
generation: 8,
|
||||
membership: {
|
||||
existingOnly: [], migrationOnly: ['production-gce-c27'], general: ['production-gce-c26']
|
||||
}
|
||||
})
|
||||
await assert.rejects(committed.commitWithoutResponse('/v1/admin/admission-selector/apply', {
|
||||
v: 1,
|
||||
attemptId: config.attemptId,
|
||||
expectedGeneration: 8,
|
||||
membership: {
|
||||
existingOnly: [], migrationOnly: [], general: ['production-gce-c26', 'production-gce-c27']
|
||||
}
|
||||
}), /response lost after commit/)
|
||||
const recovered = await operateRelayAsiaAdmission(config, committed)
|
||||
assert.equal(recovered.promoted, true)
|
||||
assert.equal(recovered.generation, 9)
|
||||
})
|
||||
|
||||
test('treats an already rolled-back promotion as recovered', async () => {
|
||||
const subject = harness({
|
||||
generation: 8,
|
||||
membership: {
|
||||
existingOnly: [], migrationOnly: ['production-gce-c27'], general: ['production-gce-c26']
|
||||
}
|
||||
})
|
||||
const config = {
|
||||
environment: 'production', mode: 'recover-promotion', cells: ['production-gce-c27'],
|
||||
expectedGeneration: 8, imageDigest: digest, attemptId: 'asia_recover_after_rollback',
|
||||
token: 'not-logged'
|
||||
}
|
||||
await assert.rejects(subject.commitWithoutResponse('/v1/admin/admission-selector/apply', {
|
||||
v: 1, attemptId: config.attemptId, expectedGeneration: 8,
|
||||
membership: {
|
||||
existingOnly: [], migrationOnly: [], general: ['production-gce-c26', 'production-gce-c27']
|
||||
}
|
||||
}), /response lost after commit/)
|
||||
await subject.apply('later_rollback', {
|
||||
existingOnly: [], migrationOnly: ['production-gce-c27'], general: ['production-gce-c26']
|
||||
})
|
||||
const recovered = await operateRelayAsiaAdmission(config, subject)
|
||||
assert.equal(recovered.promoted, false)
|
||||
assert.equal(recovered.generation, 10)
|
||||
})
|
||||
|
||||
test('recovers a committed rollback when the workflow retries the original generation', async () => {
|
||||
const subject = harness({
|
||||
generation: 9,
|
||||
membership: {
|
||||
existingOnly: [], migrationOnly: [], general: ['production-gce-c26', 'production-gce-c27']
|
||||
}
|
||||
})
|
||||
const config = {
|
||||
environment: 'production', mode: 'rollback', cells: ['production-gce-c27'],
|
||||
expectedGeneration: 9, imageDigest: digest, attemptId: 'asia_rollback_retry',
|
||||
token: 'not-logged'
|
||||
}
|
||||
await assert.rejects(subject.commitWithoutResponse(
|
||||
'/v1/admin/admission-selector/apply',
|
||||
{
|
||||
v: 1,
|
||||
attemptId: config.attemptId,
|
||||
expectedGeneration: config.expectedGeneration,
|
||||
membership: {
|
||||
existingOnly: [], migrationOnly: ['production-gce-c27'],
|
||||
general: ['production-gce-c26']
|
||||
}
|
||||
}
|
||||
), /response lost after commit/)
|
||||
const recovered = await operateRelayAsiaAdmission(config, subject)
|
||||
assert.equal(recovered.recovered, true)
|
||||
assert.equal(recovered.generation, 10)
|
||||
assert.equal(recovered.states['production-gce-c27'], 'migration-only')
|
||||
})
|
||||
|
||||
test('rejects a committed transition retry after a later selector change', async () => {
|
||||
const subject = harness({
|
||||
generation: 8,
|
||||
membership: {
|
||||
existingOnly: [], migrationOnly: ['production-gce-c27'], general: ['production-gce-c26']
|
||||
}
|
||||
})
|
||||
const config = {
|
||||
environment: 'production', mode: 'promote', cells: ['production-gce-c27'],
|
||||
expectedGeneration: 8, imageDigest: digest, attemptId: 'asia_stale_promote',
|
||||
token: 'not-logged'
|
||||
}
|
||||
await assert.rejects(subject.commitWithoutResponse('/v1/admin/admission-selector/apply', {
|
||||
v: 1, attemptId: config.attemptId, expectedGeneration: 8,
|
||||
membership: {
|
||||
existingOnly: [], migrationOnly: [], general: ['production-gce-c26', 'production-gce-c27']
|
||||
}
|
||||
}), /response lost after commit/)
|
||||
await subject.apply('later_rollback', {
|
||||
existingOnly: [], migrationOnly: ['production-gce-c27'], general: ['production-gce-c26']
|
||||
})
|
||||
await assert.rejects(
|
||||
operateRelayAsiaAdmission(config, subject),
|
||||
/does not match the requested Asia transition/
|
||||
)
|
||||
})
|
||||
|
||||
test('requires the C27 canary before promoting C28 and C29', async () => {
|
||||
const subject = harness({
|
||||
generation: 8,
|
||||
membership: {
|
||||
existingOnly: [],
|
||||
migrationOnly: ['production-gce-c27', 'production-gce-c28', 'production-gce-c29'],
|
||||
general: ['production-gce-c26']
|
||||
}
|
||||
})
|
||||
await assert.rejects(operateRelayAsiaAdmission({
|
||||
environment: 'production', mode: 'promote',
|
||||
cells: ['production-gce-c28', 'production-gce-c29'], expectedGeneration: 8,
|
||||
imageDigest: digest, attemptId: 'asia_wave_before_canary', token: 'not-logged'
|
||||
}, subject), /C27 canary/)
|
||||
})
|
||||
|
||||
const launchCells = ['production-gce-c27', 'production-gce-c28', 'production-gce-c29']
|
||||
|
||||
function admissionArguments(environment, mode, cellIds) {
|
||||
return [
|
||||
'--environment', environment, '--mode', mode, '--cell-ids', cellIds,
|
||||
'--image-digest', digest, '--expected-generation', '9', '--attempt-id', 'asia_wave_9'
|
||||
]
|
||||
}
|
||||
|
||||
test('accepts only reviewed Asia admission waves', () => {
|
||||
const accepted = [
|
||||
['inspect', 'production-gce-c27,production-gce-c28,production-gce-c29'],
|
||||
['inspect', 'production-gce-c30'],
|
||||
['inspect', 'production-gce-c31'],
|
||||
['verify', 'production-gce-c27,production-gce-c28,production-gce-c29'],
|
||||
['verify', 'production-gce-c30'],
|
||||
['verify', 'production-gce-c31'],
|
||||
['initialize', 'production-gce-c27,production-gce-c28,production-gce-c29'],
|
||||
['register', 'production-gce-c27,production-gce-c28,production-gce-c29'],
|
||||
['register', 'production-gce-c30'],
|
||||
['registered', 'production-gce-c30'],
|
||||
['register', 'production-gce-c31'],
|
||||
['registered', 'production-gce-c31'],
|
||||
['promote', 'production-gce-c27'],
|
||||
['promote', 'production-gce-c28,production-gce-c29'],
|
||||
['promote', 'production-gce-c30'],
|
||||
['recover-promotion', 'production-gce-c30'],
|
||||
['promote', 'production-gce-c31'],
|
||||
['recover-promotion', 'production-gce-c31'],
|
||||
['rollback', 'production-gce-c27'],
|
||||
['rollback', 'production-gce-c28,production-gce-c29'],
|
||||
['rollback', 'production-gce-c30'],
|
||||
['rollback', 'production-gce-c31'],
|
||||
['rollback', 'production-gce-c27,production-gce-c28,production-gce-c29'],
|
||||
['rollback', 'production-gce-c27,production-gce-c28,production-gce-c29,production-gce-c30,production-gce-c31,production-gce-c32,production-gce-c33,production-gce-c34,production-gce-c35'],
|
||||
...['production-gce-c32', 'production-gce-c33'].flatMap((cellId) => [
|
||||
'inspect', 'verify', 'register', 'registered', 'promote', 'recover-promotion', 'rollback'
|
||||
].map((mode) => [mode, cellId])),
|
||||
...['inspect', 'verify', 'register', 'registered', 'promote', 'recover-promotion', 'rollback']
|
||||
.map((mode) => [mode, 'production-gce-c34']),
|
||||
...['inspect', 'verify', 'register', 'registered', 'promote', 'recover-promotion', 'rollback']
|
||||
.map((mode) => [mode, 'production-gce-c35']),
|
||||
['inspect', 'production-gce-c27,production-gce-c28,production-gce-c29,production-gce-c30,production-gce-c31,production-gce-c32,production-gce-c33,production-gce-c34,production-gce-c35']
|
||||
]
|
||||
for (const [mode, cellIds] of accepted) {
|
||||
assert.deepEqual(
|
||||
parseRelayAsiaAdmissionArguments(admissionArguments('production', mode, cellIds)).cells,
|
||||
cellIds.split(','),
|
||||
`${mode} ${cellIds}`
|
||||
)
|
||||
}
|
||||
const rejected = [
|
||||
['promote', 'production-gce-c34,production-gce-c35'],
|
||||
['promote', 'production-gce-c31,production-gce-c35'],
|
||||
['register', 'production-gce-c34,production-gce-c35'],
|
||||
['inspect', 'production-gce-c27,production-gce-c28,production-gce-c29,production-gce-c30,production-gce-c31,production-gce-c32,production-gce-c33,production-gce-c34'],
|
||||
['rollback', 'production-gce-c27,production-gce-c28,production-gce-c29,production-gce-c30,production-gce-c31,production-gce-c32,production-gce-c33,production-gce-c34'],
|
||||
['initialize', 'production-gce-c30'],
|
||||
['initialize', 'production-gce-c27,production-gce-c28,production-gce-c29,production-gce-c30,production-gce-c31'],
|
||||
['register', 'production-gce-c27,production-gce-c28,production-gce-c29,production-gce-c30,production-gce-c31'],
|
||||
['register', 'production-gce-c30,production-gce-c31'],
|
||||
['register', 'production-gce-c29,production-gce-c30'],
|
||||
['registered', 'production-gce-c27,production-gce-c28,production-gce-c29,production-gce-c30,production-gce-c31'],
|
||||
['inspect', 'production-gce-c27,production-gce-c28,production-gce-c29,production-gce-c30'],
|
||||
['inspect', 'production-gce-c27,production-gce-c28,production-gce-c29,production-gce-c30,production-gce-c31'],
|
||||
['initialize', 'production-gce-c32'],
|
||||
['register', 'production-gce-c32,production-gce-c33'],
|
||||
['register', 'production-gce-c31,production-gce-c32'],
|
||||
['promote', 'production-gce-c32,production-gce-c33'],
|
||||
['rollback', 'production-gce-c32,production-gce-c33'],
|
||||
['rollback', 'production-gce-c27,production-gce-c28,production-gce-c29,production-gce-c30,production-gce-c31'],
|
||||
['verify', 'production-gce-c30,production-gce-c31'],
|
||||
['verify', 'production-gce-c27,production-gce-c30'],
|
||||
['promote', 'production-gce-c27,production-gce-c30'],
|
||||
['promote', 'production-gce-c28,production-gce-c29,production-gce-c30'],
|
||||
['promote', 'production-gce-c30,production-gce-c31'],
|
||||
['promote', 'production-gce-c33,production-gce-c34'],
|
||||
['promote', 'production-gce-c31,production-gce-c34'],
|
||||
['register', 'production-gce-c33,production-gce-c34'],
|
||||
['inspect', 'production-gce-c27,production-gce-c28,production-gce-c29,production-gce-c30,production-gce-c31,production-gce-c32,production-gce-c33'],
|
||||
['rollback', 'production-gce-c27,production-gce-c28,production-gce-c29,production-gce-c30,production-gce-c31,production-gce-c32,production-gce-c33'],
|
||||
['rollback', 'production-gce-c27,production-gce-c30'],
|
||||
['rollback', 'production-gce-c30,production-gce-c31'],
|
||||
['rollback', 'production-gce-c27,production-gce-c28,production-gce-c29,production-gce-c30'],
|
||||
['rollback', 'production-gce-c28,production-gce-c29,production-gce-c30'],
|
||||
['rollback', 'production-gce-c29'],
|
||||
['register', 'staging-gce-c4']
|
||||
]
|
||||
for (const [mode, cellIds] of rejected) {
|
||||
assert.throws(
|
||||
() => parseRelayAsiaAdmissionArguments(admissionArguments('production', mode, cellIds)),
|
||||
/--cell-ids/,
|
||||
`${mode} ${cellIds}`
|
||||
)
|
||||
}
|
||||
assert.deepEqual(
|
||||
parseRelayAsiaAdmissionArguments(admissionArguments('staging', 'promote', 'staging-gce-c4')).cells,
|
||||
['staging-gce-c4']
|
||||
)
|
||||
assert.throws(
|
||||
() => parseRelayAsiaAdmissionArguments(admissionArguments('staging', 'promote', 'production-gce-c30')),
|
||||
/--cell-ids are invalid/
|
||||
)
|
||||
})
|
||||
|
||||
test('registers C30 alone beside the general launch cells', async () => {
|
||||
const subject = harness({
|
||||
generation: 9,
|
||||
membership: { existingOnly: [], migrationOnly: [], general: [...launchCells] }
|
||||
})
|
||||
const result = await operateRelayAsiaAdmission({
|
||||
environment: 'production', mode: 'register', cells: ['production-gce-c30'],
|
||||
expectedGeneration: 9, imageDigest: digest, attemptId: 'asia_register_c30', token: 'not-logged'
|
||||
}, subject)
|
||||
const request = subject.requests.find(({ path }) => path.endsWith('/add-migration-cells'))
|
||||
assert.deepEqual(request.body.cells, [{
|
||||
cellId: 'production-gce-c30', cellUrl: 'https://c30.relay.onorca.dev', region: 'asia-east2',
|
||||
capacityRequests: 6_000, connectionHardCap: 3_000, connectionUnobservedBound: 60
|
||||
}])
|
||||
assert.deepEqual(result.states, { 'production-gce-c30': 'migration-only' })
|
||||
assert.deepEqual(subject.selector().membership.general, launchCells)
|
||||
})
|
||||
|
||||
test('registers C31 alone beside the general C27-C30', async () => {
|
||||
const general = [...launchCells, 'production-gce-c30']
|
||||
const subject = harness({
|
||||
generation: 11,
|
||||
membership: { existingOnly: [], migrationOnly: [], general: [...general] }
|
||||
})
|
||||
const result = await operateRelayAsiaAdmission({
|
||||
environment: 'production', mode: 'register', cells: ['production-gce-c31'],
|
||||
expectedGeneration: 11, imageDigest: digest, attemptId: 'asia_register_c31', token: 'not-logged'
|
||||
}, subject)
|
||||
const request = subject.requests.find(({ path }) => path.endsWith('/add-migration-cells'))
|
||||
assert.deepEqual(request.body.cells, [{
|
||||
cellId: 'production-gce-c31', cellUrl: 'https://c31.relay.onorca.dev', region: 'asia-east2',
|
||||
capacityRequests: 6_000, connectionHardCap: 3_000, connectionUnobservedBound: 60
|
||||
}])
|
||||
assert.deepEqual(result.states, { 'production-gce-c31': 'migration-only' })
|
||||
assert.deepEqual(subject.selector().membership.general, general)
|
||||
})
|
||||
|
||||
test('registers the C34 spare alone as migration-only in asia-east2', async () => {
|
||||
const general = [...launchCells, 'production-gce-c30', 'production-gce-c31']
|
||||
const subject = harness({
|
||||
generation: 17,
|
||||
membership: {
|
||||
existingOnly: [],
|
||||
migrationOnly: [],
|
||||
general: [...general, 'production-gce-c32', 'production-gce-c33']
|
||||
}
|
||||
})
|
||||
const result = await operateRelayAsiaAdmission({
|
||||
environment: 'production', mode: 'register', cells: ['production-gce-c34'],
|
||||
expectedGeneration: 17, imageDigest: digest, attemptId: 'asia_register_c34', token: 'not-logged'
|
||||
}, subject)
|
||||
const request = subject.requests.find(({ path }) => path.endsWith('/add-migration-cells'))
|
||||
assert.deepEqual(request.body.cells, [{
|
||||
cellId: 'production-gce-c34', cellUrl: 'https://c34.relay.onorca.dev', region: 'asia-east2',
|
||||
capacityRequests: 6_000, connectionHardCap: 3_000, connectionUnobservedBound: 60
|
||||
}])
|
||||
assert.deepEqual(result.states, { 'production-gce-c34': 'migration-only' })
|
||||
assert.equal(subject.selector().membership.general.includes('production-gce-c34'), false)
|
||||
})
|
||||
|
||||
test('registers the C35 spare alone as migration-only in asia-east2', async () => {
|
||||
const general = [...launchCells, 'production-gce-c30', 'production-gce-c31']
|
||||
const subject = harness({
|
||||
generation: 17,
|
||||
membership: {
|
||||
existingOnly: [],
|
||||
migrationOnly: [],
|
||||
general: [...general, 'production-gce-c32', 'production-gce-c33', 'production-gce-c34']
|
||||
}
|
||||
})
|
||||
const result = await operateRelayAsiaAdmission({
|
||||
environment: 'production', mode: 'register', cells: ['production-gce-c35'],
|
||||
expectedGeneration: 17, imageDigest: digest, attemptId: 'asia_register_c35', token: 'not-logged'
|
||||
}, subject)
|
||||
const request = subject.requests.find(({ path }) => path.endsWith('/add-migration-cells'))
|
||||
assert.deepEqual(request.body.cells, [{
|
||||
cellId: 'production-gce-c35', cellUrl: 'https://c35.relay.onorca.dev', region: 'asia-east2',
|
||||
capacityRequests: 6_000, connectionHardCap: 3_000, connectionUnobservedBound: 60
|
||||
}])
|
||||
assert.deepEqual(result.states, { 'production-gce-c35': 'migration-only' })
|
||||
assert.equal(subject.selector().membership.general.includes('production-gce-c35'), false)
|
||||
})
|
||||
|
||||
test('promotes the registered C34 spare to general and leaves every other cell alone', async () => {
|
||||
const general = [
|
||||
...launchCells, 'production-gce-c30', 'production-gce-c31', 'production-gce-c32', 'production-gce-c33'
|
||||
]
|
||||
const membership = {
|
||||
existingOnly: [],
|
||||
migrationOnly: ['production-gce-c17', 'production-gce-c34'],
|
||||
general: [...general]
|
||||
}
|
||||
const subject = harness({ generation: 345, membership })
|
||||
const result = await operateRelayAsiaAdmission({
|
||||
environment: 'production', mode: 'promote', cells: ['production-gce-c34'],
|
||||
expectedGeneration: 345, imageDigest: digest, attemptId: 'asia_promote_c34', token: 'not-logged'
|
||||
}, subject)
|
||||
assert.deepEqual(result.states, { 'production-gce-c34': 'general' })
|
||||
assert.deepEqual(subject.selector().membership.migrationOnly, ['production-gce-c17'])
|
||||
assert.deepEqual(subject.selector().membership.general, [...general, 'production-gce-c34'].sort())
|
||||
})
|
||||
|
||||
test('promotes the registered C35 spare to general and leaves every other cell alone', async () => {
|
||||
const general = [
|
||||
...launchCells, 'production-gce-c30', 'production-gce-c31', 'production-gce-c32', 'production-gce-c33',
|
||||
'production-gce-c34'
|
||||
]
|
||||
const membership = {
|
||||
existingOnly: [],
|
||||
migrationOnly: ['production-gce-c17', 'production-gce-c35'],
|
||||
general: [...general]
|
||||
}
|
||||
const subject = harness({ generation: 345, membership })
|
||||
const result = await operateRelayAsiaAdmission({
|
||||
environment: 'production', mode: 'promote', cells: ['production-gce-c35'],
|
||||
expectedGeneration: 345, imageDigest: digest, attemptId: 'asia_promote_c35', token: 'not-logged'
|
||||
}, subject)
|
||||
assert.deepEqual(result.states, { 'production-gce-c35': 'general' })
|
||||
assert.deepEqual(subject.selector().membership.migrationOnly, ['production-gce-c17'])
|
||||
assert.deepEqual(subject.selector().membership.general, [...general, 'production-gce-c35'].sort())
|
||||
})
|
||||
|
||||
const usRegions = { 'production-gce-c32': 'us-central1', 'production-gce-c33': 'us-central1' }
|
||||
|
||||
test('registers C32 and C33 one at a time in us-central1 at the Asia shape', async () => {
|
||||
const general = [...launchCells, 'production-gce-c30', 'production-gce-c31']
|
||||
for (const [cellId, generation] of [['production-gce-c32', 13], ['production-gce-c33', 15]]) {
|
||||
const hostname = cellId.split('-').at(-1)
|
||||
const subject = harness({
|
||||
generation, membership: { existingOnly: [], migrationOnly: [], general: [...general] }
|
||||
}, {}, usRegions)
|
||||
const result = await operateRelayAsiaAdmission({
|
||||
environment: 'production', mode: 'register', cells: [cellId],
|
||||
expectedGeneration: generation, imageDigest: digest, attemptId: `us_register_${hostname}`,
|
||||
token: 'not-logged'
|
||||
}, subject)
|
||||
const request = subject.requests.find(({ path }) => path.endsWith('/add-migration-cells'))
|
||||
assert.deepEqual(request.body.cells, [{
|
||||
cellId, cellUrl: `https://${hostname}.relay.onorca.dev`, region: 'us-central1',
|
||||
capacityRequests: 6_000, connectionHardCap: 3_000, connectionUnobservedBound: 60
|
||||
}])
|
||||
assert.deepEqual(result.states, { [cellId]: 'migration-only' })
|
||||
// A US cell whose runtime reports Asia is the wrong cell, not a US one.
|
||||
await assert.rejects(operateRelayAsiaAdmission({
|
||||
environment: 'production', mode: 'register', cells: [cellId],
|
||||
expectedGeneration: generation, imageDigest: digest, attemptId: `us_register_${hostname}`,
|
||||
token: 'not-logged'
|
||||
}, harness({
|
||||
generation, membership: { existingOnly: [], migrationOnly: [], general: [...general] }
|
||||
})), new RegExp(`${cellId} runtime does not match`))
|
||||
}
|
||||
})
|
||||
|
||||
test('promotes a US cell without the Asia launch order, which still binds Asia cells', async () => {
|
||||
const selector = (cellId) => ({
|
||||
generation: 14,
|
||||
membership: {
|
||||
existingOnly: [],
|
||||
migrationOnly: [cellId, 'production-gce-c27', 'production-gce-c28'].sort(),
|
||||
general: ['production-gce-c29']
|
||||
}
|
||||
})
|
||||
const config = (cellId) => ({
|
||||
environment: 'production', mode: 'promote', cells: [cellId],
|
||||
expectedGeneration: 14, imageDigest: digest, attemptId: 'us_promote_wave', token: 'not-logged'
|
||||
})
|
||||
const result = await operateRelayAsiaAdmission(
|
||||
config('production-gce-c32'), harness(selector('production-gce-c32'), {}, usRegions)
|
||||
)
|
||||
assert.deepEqual(result.states, { 'production-gce-c32': 'general' })
|
||||
await assert.rejects(
|
||||
operateRelayAsiaAdmission(config('production-gce-c31'), harness(selector('production-gce-c31'))),
|
||||
/C27 canary/
|
||||
)
|
||||
})
|
||||
|
||||
test('requires the C27 canary to be general before promoting C30', async () => {
|
||||
const selector = (general) => ({
|
||||
generation: 10,
|
||||
membership: {
|
||||
existingOnly: [],
|
||||
migrationOnly: ['production-gce-c30', ...launchCells.filter((cell) => !general.includes(cell))].sort(),
|
||||
general
|
||||
}
|
||||
})
|
||||
const config = {
|
||||
environment: 'production', mode: 'promote', cells: ['production-gce-c30'],
|
||||
expectedGeneration: 10, imageDigest: digest, attemptId: 'asia_promote_c30', token: 'not-logged'
|
||||
}
|
||||
await assert.rejects(
|
||||
operateRelayAsiaAdmission(config, harness(selector(['production-gce-c28', 'production-gce-c29']))),
|
||||
/C27 canary/
|
||||
)
|
||||
await assert.rejects(
|
||||
operateRelayAsiaAdmission(config, harness(selector(['production-gce-c27', 'production-gce-c29']))),
|
||||
/every launch cell to be general/
|
||||
)
|
||||
const subject = harness(selector([...launchCells]))
|
||||
const result = await operateRelayAsiaAdmission(config, subject)
|
||||
assert.deepEqual(result.states, { 'production-gce-c30': 'general' })
|
||||
assert.equal(subject.requests.filter(({ path }) => path === '/v1/admin/cell-status').length, 1)
|
||||
})
|
||||
|
||||
test('promotes C30 on its own digest while the launch cells serve another', async () => {
|
||||
const c30Digest = `sha256:${'b'.repeat(64)}`
|
||||
const selector = {
|
||||
generation: 10,
|
||||
membership: { existingOnly: [], migrationOnly: ['production-gce-c30'], general: [...launchCells] }
|
||||
}
|
||||
const config = {
|
||||
environment: 'production', mode: 'promote', cells: ['production-gce-c30'],
|
||||
expectedGeneration: 10, imageDigest: c30Digest, attemptId: 'asia_promote_c30', token: 'not-logged'
|
||||
}
|
||||
const digests = { 'production-gce-c30': c30Digest }
|
||||
const result = await operateRelayAsiaAdmission(config, harness(selector, digests))
|
||||
assert.deepEqual(result.states, { 'production-gce-c30': 'general' })
|
||||
await assert.rejects(
|
||||
operateRelayAsiaAdmission({ ...config, imageDigest: digest }, harness(selector, digests)),
|
||||
/production-gce-c30 runtime does not match/
|
||||
)
|
||||
})
|
||||
|
||||
const admissionWorkflow = readRelayWorkflow('operate-relay-asia-admission.yml')
|
||||
|
||||
function workflowBlock(first, last) {
|
||||
const start = admissionWorkflow.indexOf(first)
|
||||
const end = admissionWorkflow.indexOf(last, start)
|
||||
assert.ok(start !== -1 && end !== -1, first)
|
||||
return admissionWorkflow.slice(start, end + last.length).replace(/^ {10}/gm, '')
|
||||
}
|
||||
|
||||
// Runs the workflow's own promotion input block, so each wave's canary routing is what ships.
|
||||
function promotionOutputs(cellIds) {
|
||||
const temp = mkdtempSync(join(tmpdir(), 'relay-admission-inputs-'))
|
||||
try {
|
||||
const output = join(temp, 'output')
|
||||
const result = spawnSync('bash', ['-c', workflowBlock(
|
||||
'set -euo pipefail\n test -n "${DEPLOY_WORKLOAD_IDENTITY_PROVIDER}"',
|
||||
'} >> "${GITHUB_OUTPUT}"'
|
||||
)], {
|
||||
env: {
|
||||
...process.env, GITHUB_OUTPUT: output, DEPLOY_WORKLOAD_IDENTITY_PROVIDER: 'provider',
|
||||
DEPLOY_SERVICE_ACCOUNT: 'account', IMAGE_DIGEST: digest, OPERATION_MODE: 'promote',
|
||||
EXPECTED_SELECTOR_GENERATION: '9', SELECTOR_ATTEMPT_ID: 'promote_wave_9',
|
||||
OPERATION_CONFIRMATION: 'PROMOTE_ASIA_GENERAL', TARGET_ENVIRONMENT: 'production',
|
||||
TARGET_CELL_IDS: cellIds, EXPECTED_SELECTOR_MEMBERSHIP_SHA256: '', DIRECTOR_IMAGE_DIGEST: '',
|
||||
EVIDENCE_RUN_ID: '', EVIDENCE_RUN_ATTEMPT: ''
|
||||
},
|
||||
encoding: 'utf8'
|
||||
})
|
||||
if (result.status !== 0) return null
|
||||
return Object.fromEntries(readFileSync(output, 'utf8').trim().split('\n')
|
||||
.map((line) => line.split('=')))
|
||||
} finally {
|
||||
rmSync(temp, { recursive: true, force: true })
|
||||
}
|
||||
}
|
||||
|
||||
test('runs each later cell\'s own canary with load aimed at that cell\'s region', () => {
|
||||
for (const [cellId, region] of [
|
||||
['production-gce-c30', 'asia-east2'], ['production-gce-c31', 'asia-east2'],
|
||||
['production-gce-c32', 'us-central1'], ['production-gce-c33', 'us-central1'],
|
||||
['production-gce-c34', 'asia-east2'], ['production-gce-c35', 'asia-east2']
|
||||
]) {
|
||||
const outputs = promotionOutputs(cellId)
|
||||
assert.equal(outputs?.canary, 'true', cellId)
|
||||
assert.equal(outputs.canary_cell, cellId)
|
||||
assert.equal(outputs.canary_region, region, cellId)
|
||||
assert.equal(outputs.evidence_kind, 'none')
|
||||
}
|
||||
assert.equal(promotionOutputs('production-gce-c32,production-gce-c33'), null)
|
||||
assert.equal(promotionOutputs('production-gce-c33,production-gce-c34'), null)
|
||||
assert.equal(promotionOutputs('production-gce-c34,production-gce-c35'), null)
|
||||
const canaryStart = workflowBlock('case "${CANARY_CELL}" in', '\n esac')
|
||||
for (const cellId of ['production-gce-c32', 'production-gce-c33', 'production-gce-c34', 'production-gce-c35']) {
|
||||
const result = spawnSync('bash', ['-euo', 'pipefail', '-c',
|
||||
`${canaryStart}\necho "\${verify_cells} \${expected_states}"`], {
|
||||
env: { ...process.env, CANARY_CELL: cellId }, encoding: 'utf8'
|
||||
})
|
||||
assert.equal(result.stdout.trim(), `${cellId} {"${cellId}":"general"}`)
|
||||
}
|
||||
assert.match(admissionWorkflow, /--preferred-region "\$\{CANARY_REGION\}"/)
|
||||
assert.match(admissionWorkflow, /CANARY_REGION: \$\{\{ steps\.inputs\.outputs\.canary_region \}\}/)
|
||||
})
|
||||
|
||||
const stagingDeadCellWave = ['staging-gce-c5', 'staging-gce-c6', 'staging-gce-c7']
|
||||
const stagingUsShape = Object.fromEntries(stagingDeadCellWave.map((cellId) => [cellId, 'us-central1']))
|
||||
const stagingUsCaps = Object.fromEntries(stagingDeadCellWave.map((cellId) => [cellId, 600]))
|
||||
|
||||
test('registers the staging dead-cell wave at its own US shape', async () => {
|
||||
const subject = harness({
|
||||
generation: 43,
|
||||
membership: { existingOnly: ['staging-gce-c3'], migrationOnly: [], general: ['staging-gce-c1', 'staging-gce-c2', 'staging-gce-c4'] }
|
||||
}, {}, stagingUsShape, stagingUsCaps)
|
||||
const result = await operateRelayAsiaAdmission({
|
||||
environment: 'staging', mode: 'register', cells: stagingDeadCellWave,
|
||||
expectedGeneration: 43, imageDigest: digest, attemptId: 'staging_c5_c7_register', token: 'not-logged'
|
||||
}, subject)
|
||||
const request = subject.requests.find(({ path }) => path.endsWith('/add-migration-cells'))
|
||||
assert.deepEqual(request.body.cells.map((cell) => cell.cellUrl), [
|
||||
'https://c5.relay-staging.onorca.dev', 'https://c6.relay-staging.onorca.dev', 'https://c7.relay-staging.onorca.dev'
|
||||
])
|
||||
assert.ok(request.body.cells.every((cell) =>
|
||||
cell.region === 'us-central1' && cell.capacityRequests === 4_000 &&
|
||||
cell.connectionHardCap === 600 && cell.connectionUnobservedBound === 60
|
||||
))
|
||||
assert.deepEqual(new Set(Object.values(result.states)), new Set(['migration-only']))
|
||||
})
|
||||
|
||||
test('promotes the staging dead-cell wave through the general route, not the C4 proof route', async () => {
|
||||
const subject = harness({
|
||||
generation: 44,
|
||||
membership: {
|
||||
existingOnly: ['staging-gce-c3'],
|
||||
migrationOnly: [...stagingDeadCellWave],
|
||||
general: ['staging-gce-c1', 'staging-gce-c2', 'staging-gce-c4']
|
||||
}
|
||||
}, {}, stagingUsShape, stagingUsCaps)
|
||||
const result = await operateRelayAsiaAdmission({
|
||||
environment: 'staging', mode: 'promote', cells: stagingDeadCellWave,
|
||||
expectedGeneration: 44, imageDigest: digest, attemptId: 'staging_c5_c7_promote', token: 'not-logged'
|
||||
}, subject)
|
||||
assert.ok(subject.requests.some(({ path }) => path === '/v1/admin/admission-selector/apply'))
|
||||
assert.ok(!subject.requests.some(({ path }) => path.endsWith('/apply-staging-asia-proof')))
|
||||
assert.deepEqual(new Set(Object.values(result.states)), new Set(['general']))
|
||||
assert.deepEqual(subject.selector().membership.general, [
|
||||
'staging-gce-c1', 'staging-gce-c2', 'staging-gce-c4', ...stagingDeadCellWave
|
||||
])
|
||||
})
|
||||
|
||||
test('refuses a staging wave that was not reviewed', () => {
|
||||
assert.throws(() => parseRelayAsiaAdmissionArguments([
|
||||
'--environment', 'staging', '--mode', 'promote', '--cell-ids', 'staging-gce-c4,staging-gce-c5',
|
||||
'--expected-generation', '44', '--attempt-id', 'staging_mixed_wave', '--image-digest', digest
|
||||
]), /reviewed admission wave/)
|
||||
assert.throws(() => parseRelayAsiaAdmissionArguments([
|
||||
'--environment', 'staging', '--mode', 'register', '--cell-ids', 'staging-gce-c8',
|
||||
'--expected-generation', '44', '--attempt-id', 'staging_unknown', '--image-digest', digest
|
||||
]), /cell-ids are invalid/)
|
||||
})
|
||||
@@ -1,619 +0,0 @@
|
||||
#!/usr/bin/env node
|
||||
// Writes one cell's runtime switch object (cells/<cellId>.json in <project>-relay-control) and
|
||||
// reads back what the cell applied. One cell per run by construction: the object is per cell.
|
||||
//
|
||||
// dry-run: checks the cell and the current generation, prints the change, writes nothing.
|
||||
// write: the same checks, then a compare-and-swap write (ifGenerationMatch), then waits for
|
||||
// the cell's runtime status to report the new generation applied.
|
||||
import { parseArgs } from 'node:util'
|
||||
import { pathToFileURL } from 'node:url'
|
||||
import { fetchAdminOnceMore } from './relay-admin-transient-retry.mjs'
|
||||
|
||||
export const CELL_ID_PATTERN = /^(production|staging)-gce-c[0-9]{1,3}$/
|
||||
export const WRITE_CONFIRMATION = 'WRITE_RELAY_CELL_FLAGS'
|
||||
// The cell polls every 5 s (+20% jitter); 15 s leaves two polls of slack.
|
||||
export const READ_BACK_TIMEOUT_MS = 15_000
|
||||
const READ_BACK_INTERVAL_MS = 1_000
|
||||
const STORAGE_ROOT = 'https://storage.googleapis.com'
|
||||
|
||||
// Every switch a cell image may know, with what an absent key means on the cell. A cell
|
||||
// resets a key missing from its object to this default, so a write must carry every key it
|
||||
// does not mean to change: the tool reads the current object and changes only what is named.
|
||||
const enumOf = (...options) => (value) => (options.includes(value) ? value : undefined)
|
||||
const booleanValue = (value) => (value === 'true' ? true : value === 'false' ? false : undefined)
|
||||
const numberIn = (min, max, integer) => (value) => {
|
||||
const number = Number(value)
|
||||
if (value === '' || !Number.isFinite(number) || number < min || number > max) return undefined
|
||||
if (integer && !Number.isInteger(number)) return undefined
|
||||
return number
|
||||
}
|
||||
export const CELL_FLAG_SPECS = {
|
||||
readinessLocal: { parse: booleanValue, default: false },
|
||||
ticketCheck: { parse: enumOf('off', 'shadow', 'enforce'), default: 'off' },
|
||||
admitMode: { parse: enumOf('db', 'reserve'), default: 'db' },
|
||||
// Absent means the cell's region default.
|
||||
intakePerSec: {
|
||||
parse: (value) => {
|
||||
const rate = numberIn(0, 1_000, false)(value)
|
||||
return rate === 0 ? undefined : rate
|
||||
}
|
||||
},
|
||||
reserveDryRun: { parse: booleanValue, default: false },
|
||||
rejectionFence: { parse: booleanValue, default: true },
|
||||
// Absent keeps the cell's boot value.
|
||||
readTimeoutMarginMs: { parse: numberIn(1_000, 60_000, true) },
|
||||
// Flip-back leases in flight; absent is a third of the cell's pool.
|
||||
reregisterInFlight: { parse: numberIn(1, 16, true) },
|
||||
// Step 6: acts only while the cell is reserve.
|
||||
phonePath: { parse: enumOf('db', 'memory'), default: 'db' }
|
||||
}
|
||||
|
||||
export const RESERVE_CONFIRMATION = 'RESERVE'
|
||||
export const BREAK_GLASS_CONFIRMATION = 'BREAK GLASS'
|
||||
// Longer than a director's cached read of the admit-mode table (5 s).
|
||||
export const RESERVE_MODE_CACHE_WAIT_MS = 10_000
|
||||
// How long a flip back may take to lease every control the cell admitted from memory: its
|
||||
// control count at a per-lane pace below the measured 0.38/s (170 ms trips), within the job.
|
||||
export const REREGISTER_WAIT_MIN_MS = 8 * 60_000
|
||||
export const REREGISTER_WAIT_MAX_MS = 50 * 60_000
|
||||
const REREGISTER_CONTROLS_PER_LANE_PER_SEC = 0.3
|
||||
// The cell takes a third of its database pool, and caps the switch at the pool less two
|
||||
// (host-session-registry.ts reregisterInFlightLimit). An image from before runtime-status
|
||||
// reported the pool falls back to the deployed sizes (US 10, Asia 16).
|
||||
const CELL_DATABASE_POOL_MAX = { 'asia-east2': 16 }
|
||||
const CELL_DATABASE_POOL_MAX_DEFAULT = 10
|
||||
|
||||
export function reregisterLanes(runtime) {
|
||||
const reported = runtime?.databasePoolMax
|
||||
const pool =
|
||||
Number.isInteger(reported) && reported > 0
|
||||
? reported
|
||||
: (CELL_DATABASE_POOL_MAX[runtime?.region] ?? CELL_DATABASE_POOL_MAX_DEFAULT)
|
||||
const flag = runtime?.flagsApplied?.flags?.reregisterInFlight
|
||||
if (flag === undefined) return Math.max(1, Math.floor(pool / 3))
|
||||
return Math.max(1, Math.min(Number(flag), pool - 2))
|
||||
}
|
||||
|
||||
export function reregisterWaitMs(runtime) {
|
||||
const controls = Number(runtime?.runtime?.controls ?? 0)
|
||||
const lanes = reregisterLanes(runtime)
|
||||
const estimate = (controls / (lanes * REREGISTER_CONTROLS_PER_LANE_PER_SEC)) * 1_000
|
||||
return Math.min(REREGISTER_WAIT_MAX_MS, Math.max(REREGISTER_WAIT_MIN_MS, Math.ceil(estimate)))
|
||||
}
|
||||
|
||||
// An image from before `supportedFlags` accepts the keys its applied flags list, and only
|
||||
// ticketCheck off/shadow: anything else is dropped there, or voids the whole object.
|
||||
function legacySupportedFlags(appliedFlags) {
|
||||
return Object.fromEntries(
|
||||
Object.entries(appliedFlags ?? {}).map(([key, value]) => [
|
||||
key,
|
||||
key === 'ticketCheck'
|
||||
? { type: 'enum', values: ['off', 'shadow'] }
|
||||
: { type: typeof value === 'boolean' ? 'boolean' : typeof value === 'number' ? 'number' : 'enum', values: [value] }
|
||||
])
|
||||
)
|
||||
}
|
||||
|
||||
function accepts(spec, value) {
|
||||
if (spec?.type === 'boolean') return typeof value === 'boolean'
|
||||
if (spec?.type === 'enum') return spec.values.includes(value)
|
||||
if (spec?.type === 'number') {
|
||||
return (
|
||||
typeof value === 'number' &&
|
||||
(spec.min === undefined || value >= spec.min) &&
|
||||
(spec.max === undefined || value <= spec.max) &&
|
||||
(!spec.integer || Number.isInteger(value))
|
||||
)
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// The whole object as it will be written, not only the change: a bad key or value already in
|
||||
// the file would void or be dropped on every later write too.
|
||||
export function assertSupportedObject(flags, runtime) {
|
||||
const supported = runtime.supportedFlags ?? legacySupportedFlags(runtime.flagsApplied?.flags)
|
||||
for (const [key, value] of Object.entries(flags)) {
|
||||
if (!accepts(supported[key], value)) {
|
||||
throw new Error(
|
||||
`the cell's image does not support ${key}=${JSON.stringify(value)}; roll the image first`
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A reserve cell's dead-man is fed only by director polls through the load balancer. With SQL
|
||||
// readiness the balancer drops the cell in a database outage, the polls fail, and the cell trips
|
||||
// (staging drill 2026-10-11), so reserve requires readinessLocal=true in the object it writes.
|
||||
export function assertReserveKeepsLocalReadiness(flags, changes) {
|
||||
if ((flags.admitMode ?? 'db') !== 'reserve' || flags.readinessLocal === true) return
|
||||
if ('readinessLocal' in changes && !('admitMode' in changes)) {
|
||||
throw new Error(
|
||||
'readinessLocal must stay true while the cell is reserve; flip it back with admitMode=db first, or in the same write'
|
||||
)
|
||||
}
|
||||
throw new Error(
|
||||
'admitMode=reserve requires readinessLocal=true on the cell, already written or in the same write ' +
|
||||
'(e.g. --set readinessLocal=true,admitMode=reserve): with SQL readiness the load balancer drops the cell ' +
|
||||
"in a database outage and the directors' polls can no longer feed its dead-man"
|
||||
)
|
||||
}
|
||||
|
||||
// The cell ignores phonePath=memory outside reserve, so writing it there only arms a later
|
||||
// admitMode=reserve flip to turn phones over unnamed. Rolling back with admitMode=db stays allowed.
|
||||
export function assertPhonePathNeedsReserve(flags, changes) {
|
||||
if ((flags.phonePath ?? 'db') !== 'memory') return
|
||||
if ((flags.admitMode ?? 'db') === 'reserve') {
|
||||
if (changes.admitMode === 'reserve' && !('phonePath' in changes)) {
|
||||
throw new Error(
|
||||
'the object still says phonePath=memory: name phonePath in the same write ' +
|
||||
'(e.g. --set admitMode=reserve,phonePath=default), so a reserve flip never turns phones over unnamed'
|
||||
)
|
||||
}
|
||||
return
|
||||
}
|
||||
if ('phonePath' in changes) {
|
||||
throw new Error("phonePath=memory requires admitMode=reserve on the cell, already written or in the same write")
|
||||
}
|
||||
}
|
||||
|
||||
// `key=value,key=value`; `key=default` removes the key so the cell uses its default.
|
||||
export function parseFlagChanges(text) {
|
||||
const changes = {}
|
||||
for (const part of (text ?? '').split(',').map((entry) => entry.trim()).filter(Boolean)) {
|
||||
const separator = part.indexOf('=')
|
||||
const key = separator < 0 ? part : part.slice(0, separator)
|
||||
const raw = separator < 0 ? '' : part.slice(separator + 1).trim()
|
||||
const spec = CELL_FLAG_SPECS[key]
|
||||
if (!spec) throw new Error(`unknown switch ${key}`)
|
||||
if (key in changes) throw new Error(`switch ${key} named twice`)
|
||||
if (raw === 'default') {
|
||||
changes[key] = null
|
||||
continue
|
||||
}
|
||||
const value = spec.parse(raw)
|
||||
if (value === undefined) throw new Error(`switch ${key} cannot be ${JSON.stringify(raw)}`)
|
||||
// phonePath=db is the default, stored as no key: an image without the switch refuses the key.
|
||||
changes[key] = key === 'phonePath' && value === 'db' ? null : value
|
||||
}
|
||||
if (Object.keys(changes).length === 0) throw new Error('set names no switch to change')
|
||||
return changes
|
||||
}
|
||||
|
||||
// The switches as the cell will apply them: every known key, absent ones at their default.
|
||||
function effectiveFlags(flags) {
|
||||
return Object.fromEntries(
|
||||
Object.entries(CELL_FLAG_SPECS).map(([key, spec]) => [key, flags?.[key] ?? spec.default])
|
||||
)
|
||||
}
|
||||
|
||||
export function parseCellFlagsRequest(values) {
|
||||
const mode = values.mode
|
||||
if (mode !== 'dry-run' && mode !== 'write') throw new Error('mode must be dry-run or write')
|
||||
const cellId = values['cell-id'] ?? ''
|
||||
if (!CELL_ID_PATTERN.test(cellId)) throw new Error('cell id is not a relay GCE cell id')
|
||||
const changes = parseFlagChanges(values.set)
|
||||
// 0 means "no object yet": the write then creates it and refuses if one appeared.
|
||||
const expectedGeneration = values['expected-generation'] ?? ''
|
||||
if (!/^(0|[1-9][0-9]{0,18})$/.test(expectedGeneration)) {
|
||||
throw new Error('expected-generation must be 0 or the exact current object generation')
|
||||
}
|
||||
if (mode === 'write' && values.confirmation !== `${WRITE_CONFIRMATION} ${cellId}`) {
|
||||
throw new Error(`write requires the confirmation "${WRITE_CONFIRMATION} ${cellId}"`)
|
||||
}
|
||||
// Reserve moves this cell's hosts off the database: it is typed for this cell, every time.
|
||||
if (
|
||||
mode === 'write' &&
|
||||
changes.admitMode === 'reserve' &&
|
||||
values['confirm-reserve'] !== `${RESERVE_CONFIRMATION} ${cellId}`
|
||||
) {
|
||||
throw new Error(`admitMode=reserve requires --confirm-reserve "${RESERVE_CONFIRMATION} ${cellId}"`)
|
||||
}
|
||||
// Break glass: a dead or unreachable reserve cell cannot report db, so nothing would ever
|
||||
// let the sweeps re-place its hosts. Only admitMode=db, typed for this cell.
|
||||
const breakGlass = values['break-glass'] ?? ''
|
||||
if (breakGlass !== '') {
|
||||
if (breakGlass !== `${BREAK_GLASS_CONFIRMATION} ${cellId}`) {
|
||||
throw new Error(`break glass requires --break-glass "${BREAK_GLASS_CONFIRMATION} ${cellId}"`)
|
||||
}
|
||||
if (mode !== 'write' || changes.admitMode !== null || Object.keys(changes).length !== 1) {
|
||||
throw new Error('break glass only writes admitMode=default (db), on its own')
|
||||
}
|
||||
}
|
||||
const directorOrigin = values['director-origin'] ?? ''
|
||||
if (changes.admitMode !== undefined && !/^https:\/\/relay(-staging)?\.onorca\.dev$/.test(directorOrigin)) {
|
||||
throw new Error('an admitMode change needs --director-origin, where Postgres records it')
|
||||
}
|
||||
const projectId = values['project-id'] ?? ''
|
||||
if (!/^[a-z][a-z0-9-]{4,62}$/.test(projectId)) throw new Error('project id is invalid')
|
||||
const cellOrigin = values['cell-origin'] ?? ''
|
||||
if (!/^https:\/\/c[0-9]{1,3}\.relay(-staging)?\.onorca\.dev$/.test(cellOrigin)) {
|
||||
throw new Error('cell origin is not a relay cell origin')
|
||||
}
|
||||
if (new URL(cellOrigin).hostname.split('.')[0] !== cellId.replace(/^[a-z]+-gce-/, '')) {
|
||||
throw new Error('cell origin names another cell')
|
||||
}
|
||||
return {
|
||||
mode,
|
||||
cellId,
|
||||
cellOrigin,
|
||||
directorOrigin,
|
||||
breakGlass: breakGlass !== '',
|
||||
projectId,
|
||||
expectedGeneration,
|
||||
changes
|
||||
}
|
||||
}
|
||||
|
||||
// Read-modify-write: the current object's switches with only the named ones changed.
|
||||
export function desiredObject(request, current) {
|
||||
if (current !== null && (current?.v !== 1 || current?.cellId !== request.cellId)) {
|
||||
throw new Error('the current object is not this cell\'s v1 switch object; fix it by hand')
|
||||
}
|
||||
const flags = { ...(current?.flags ?? {}) }
|
||||
for (const [key, value] of Object.entries(request.changes)) {
|
||||
if (value === null) delete flags[key]
|
||||
else flags[key] = value
|
||||
}
|
||||
return { v: 1, cellId: request.cellId, flags }
|
||||
}
|
||||
|
||||
function objectUrl(request, suffix = '') {
|
||||
const bucket = `${request.projectId}-relay-control`
|
||||
return `${STORAGE_ROOT}/storage/v1/b/${bucket}/o/${encodeURIComponent(
|
||||
`cells/${request.cellId}.json`
|
||||
)}${suffix}`
|
||||
}
|
||||
|
||||
// One transient 5xx or network failure is retried once, as every relay admin step does.
|
||||
export async function readRuntime(fetchImpl, request, idToken, wait) {
|
||||
const response = await fetchAdminOnceMore(
|
||||
fetchImpl,
|
||||
`${request.cellOrigin}/v1/admin/runtime-status`,
|
||||
{
|
||||
method: 'POST',
|
||||
headers: { authorization: `Bearer ${idToken}`, 'content-type': 'application/json' },
|
||||
body: JSON.stringify({ v: 1 })
|
||||
},
|
||||
{ wait, timeoutMs: 10_000, retryDelayMs: 1_000 }
|
||||
)
|
||||
if (!response.ok) {
|
||||
await response.body?.cancel().catch(() => undefined)
|
||||
throw new Error(`cell runtime status returned ${response.status}`)
|
||||
}
|
||||
return await response.json()
|
||||
}
|
||||
|
||||
export async function readCurrentObject(fetchImpl, request, accessToken) {
|
||||
const headers = { authorization: `Bearer ${accessToken}` }
|
||||
const metadata = await fetchImpl(objectUrl(request), { headers })
|
||||
if (metadata.status === 404) {
|
||||
await metadata.body?.cancel().catch(() => undefined)
|
||||
return { generation: '0', object: null }
|
||||
}
|
||||
if (!metadata.ok) throw new Error(`object metadata read returned ${metadata.status}`)
|
||||
const { generation } = await metadata.json()
|
||||
const media = await fetchImpl(objectUrl(request, `?alt=media&ifGenerationMatch=${generation}`), {
|
||||
headers
|
||||
})
|
||||
if (!media.ok) throw new Error(`object read returned ${media.status}`)
|
||||
return { generation: String(generation), object: await media.json().catch(() => null) }
|
||||
}
|
||||
|
||||
function generationConflict(current, expected) {
|
||||
return Object.assign(new Error(`current generation is ${current}, not expected ${expected}`), {
|
||||
generationConflict: true
|
||||
})
|
||||
}
|
||||
|
||||
async function writeObject(fetchImpl, request, object, accessToken, audit) {
|
||||
const boundary = `relay-cell-flags-${Date.now()}`
|
||||
const metadata = {
|
||||
name: `cells/${request.cellId}.json`,
|
||||
contentType: 'application/json',
|
||||
cacheControl: 'no-store',
|
||||
// Who and which run, beside the generation the bucket's versioning already keeps.
|
||||
metadata: audit
|
||||
}
|
||||
const body = [
|
||||
`--${boundary}`,
|
||||
'Content-Type: application/json; charset=UTF-8',
|
||||
'',
|
||||
JSON.stringify(metadata),
|
||||
`--${boundary}`,
|
||||
'Content-Type: application/json',
|
||||
'',
|
||||
JSON.stringify(object),
|
||||
`--${boundary}--`,
|
||||
''
|
||||
].join('\r\n')
|
||||
const bucket = `${request.projectId}-relay-control`
|
||||
const response = await fetchImpl(
|
||||
`${STORAGE_ROOT}/upload/storage/v1/b/${bucket}/o?uploadType=multipart&ifGenerationMatch=${request.expectedGeneration}`,
|
||||
{
|
||||
method: 'POST',
|
||||
headers: {
|
||||
authorization: `Bearer ${accessToken}`,
|
||||
'content-type': `multipart/related; boundary=${boundary}`
|
||||
},
|
||||
body
|
||||
}
|
||||
)
|
||||
// A 4xx means nothing was written; a 5xx or a lost reply may have been.
|
||||
if (response.status === 412) {
|
||||
await response.body?.cancel().catch(() => undefined)
|
||||
throw Object.assign(new Error('the object changed since expected-generation; re-read and retry'), {
|
||||
notWritten: true,
|
||||
generationConflict: true
|
||||
})
|
||||
}
|
||||
if (!response.ok) {
|
||||
throw Object.assign(new Error(`object write returned ${response.status}`), {
|
||||
notWritten: response.status < 500
|
||||
})
|
||||
}
|
||||
return String((await response.json()).generation)
|
||||
}
|
||||
|
||||
// The census's record (relay_cell_admit_modes, through a director). Without `admitMode` it only
|
||||
// reads. Read back on every write: sweeps act on this, not on the cell's switch file.
|
||||
async function directorAdmitMode(fetchImpl, request, idToken, wait, admitMode) {
|
||||
const response = await fetchAdminOnceMore(
|
||||
fetchImpl,
|
||||
`${request.directorOrigin}/v1/admin/cell-admit-mode`,
|
||||
{
|
||||
method: 'POST',
|
||||
headers: { authorization: `Bearer ${idToken}`, 'content-type': 'application/json' },
|
||||
body: JSON.stringify({ v: 1, cellId: request.cellId, ...(admitMode ? { admitMode } : {}) })
|
||||
},
|
||||
{ wait, timeoutMs: 10_000, retryDelayMs: 1_000 }
|
||||
)
|
||||
const body = await response.json().catch(() => null)
|
||||
if (!response.ok) throw new Error(`director refused admitMode: ${response.status} ${body?.error ?? ''}`)
|
||||
if (admitMode && body?.admitMode !== admitMode) {
|
||||
throw new Error(`director recorded admitMode ${body?.admitMode}, not ${admitMode}`)
|
||||
}
|
||||
return body
|
||||
}
|
||||
|
||||
// Every known key, not just the ones this run changed: a cell that kept an old value of any
|
||||
// switch has not applied this object. A key with no fixed default (the cell reports its own
|
||||
// value) is compared only when the object names it.
|
||||
export function sameFlags(applied, desired) {
|
||||
const expected = effectiveFlags(desired)
|
||||
const actual = effectiveFlags(applied)
|
||||
return Object.entries(CELL_FLAG_SPECS).every(
|
||||
([key, spec]) =>
|
||||
(!('default' in spec) && desired?.[key] === undefined) || actual[key] === expected[key]
|
||||
)
|
||||
}
|
||||
|
||||
// Sweeps resume only once every control the cell admitted from memory holds a lease again.
|
||||
async function waitForLeasedControls(fetchImpl, request, idToken, { now, sleep, log }) {
|
||||
let waitMs
|
||||
let deadline = now() + REREGISTER_WAIT_MIN_MS
|
||||
while (now() < deadline) {
|
||||
const runtime = await readRuntime(fetchImpl, request, idToken, sleep).catch(() => null)
|
||||
// An image without the field never admitted from memory.
|
||||
if (runtime && (runtime.admitModeEffective ?? 'db') === 'db') return
|
||||
if (runtime && waitMs === undefined) {
|
||||
waitMs = reregisterWaitMs(runtime)
|
||||
deadline = now() + waitMs
|
||||
log(JSON.stringify({ event: 'orca_relay_cell_reregistration_wait', cellId: request.cellId, waitMs }))
|
||||
}
|
||||
await sleep(5_000)
|
||||
}
|
||||
log(JSON.stringify({ event: 'orca_relay_cell_reregistration_still_running', cellId: request.cellId }))
|
||||
throw new Error(
|
||||
`cell still re-registering after ${waitMs ?? REREGISTER_WAIT_MIN_MS} ms. Its connections are safe and Postgres ` +
|
||||
'keeps reserve, so sweeps stay off the cell. Run again with --set admitMode=db, which records db once ' +
|
||||
'runtime-status reports admitModeEffective=db; to speed it up, name both in one write: ' +
|
||||
'--set admitMode=db,reregisterInFlight=N'
|
||||
)
|
||||
}
|
||||
|
||||
// No cell read-back and no wait for its controls: the cell is not answering. The file says db
|
||||
// so a cell that comes back boots in db, and Postgres says db so the sweeps re-place its hosts.
|
||||
async function breakGlassToDatabase(request, dependencies, { fetchImpl, log, sleep }) {
|
||||
const { accessToken, idToken, audit } = dependencies
|
||||
// Only for a dead cell: one that answers, or still heartbeats, takes the ordinary flip back.
|
||||
const answered = await fetchImpl(`${request.cellOrigin}/v1/admin/runtime-status`, {
|
||||
method: 'POST',
|
||||
headers: { authorization: `Bearer ${idToken}`, 'content-type': 'application/json' },
|
||||
body: JSON.stringify({ v: 1 }),
|
||||
signal: AbortSignal.timeout(10_000)
|
||||
})
|
||||
.then(async (response) => {
|
||||
await response.body?.cancel().catch(() => undefined)
|
||||
return response.ok
|
||||
})
|
||||
.catch(() => false)
|
||||
if (answered) throw new Error('the cell answers runtime-status: flip it back with --set admitMode=db, not break glass')
|
||||
const recorded = await directorAdmitMode(fetchImpl, request, idToken, sleep)
|
||||
if (recorded?.heartbeatFresh !== false) {
|
||||
throw new Error(
|
||||
`the director ${recorded?.heartbeatFresh ? 'still sees heartbeats from' : 'cannot say whether it hears'} ` +
|
||||
'the cell: break glass is only for a cell that has stopped'
|
||||
)
|
||||
}
|
||||
const current = await readCurrentObject(fetchImpl, request, accessToken)
|
||||
if (current.generation !== request.expectedGeneration) {
|
||||
throw generationConflict(current.generation, request.expectedGeneration)
|
||||
}
|
||||
// phonePath goes too: a dead cell comes back with phones on the database path as well.
|
||||
const object = desiredObject({ ...request, changes: { ...request.changes, phonePath: null } }, current.object)
|
||||
const generation = await writeObject(fetchImpl, request, object, accessToken, audit)
|
||||
await directorAdmitMode(fetchImpl, request, idToken, sleep, 'db')
|
||||
const result = { event: 'orca_relay_cell_admit_mode_break_glass', cellId: request.cellId, generation, audit }
|
||||
log(JSON.stringify(result))
|
||||
return { ...result, written: true }
|
||||
}
|
||||
|
||||
export async function operateCellFlags(request, dependencies) {
|
||||
const { fetchImpl = fetch, accessToken, idToken, audit, log = console.log } = dependencies
|
||||
const now = dependencies.now ?? Date.now
|
||||
const sleep = dependencies.sleep ?? ((ms) => new Promise((resolve) => setTimeout(resolve, ms)))
|
||||
if (request.breakGlass) return await breakGlassToDatabase(request, dependencies, { fetchImpl, log, sleep })
|
||||
const runtime = await readRuntime(fetchImpl, request, idToken, sleep)
|
||||
// Proves the origin is this cell, and that its image reads switch files at all.
|
||||
if (runtime.role !== 'cell' || runtime.cellId !== request.cellId) {
|
||||
throw new Error(`origin answers as ${runtime.role}/${runtime.cellId}, not ${request.cellId}`)
|
||||
}
|
||||
if (!runtime.flagsApplied) throw new Error('cell image has no flag channel (no flagsApplied)')
|
||||
const current = await readCurrentObject(fetchImpl, request, accessToken)
|
||||
const object = desiredObject(request, current.object)
|
||||
assertSupportedObject(object.flags, runtime)
|
||||
// A tripped dead-man: any new generation that keeps reserve re-arms it, so the cell trips
|
||||
// again and its re-registration restarts. Only a write that also says db is safe.
|
||||
if (
|
||||
runtime.flagsApplied.flags?.admitMode === 'reserve' &&
|
||||
runtime.admitModeEffective === 'db' &&
|
||||
(object.flags.admitMode ?? 'db') === 'reserve'
|
||||
) {
|
||||
throw new Error(
|
||||
'the cell tripped its dead-man (reserve configured, db effective); a write that keeps reserve re-arms it. ' +
|
||||
'Name admitMode=db in the same write, e.g. --set admitMode=db,reregisterInFlight=N'
|
||||
)
|
||||
}
|
||||
assertReserveKeepsLocalReadiness(object.flags, request.changes)
|
||||
assertPhonePathNeedsReserve(object.flags, request.changes)
|
||||
const plan = {
|
||||
event: 'orca_relay_cell_flags_plan',
|
||||
mode: request.mode,
|
||||
cellId: request.cellId,
|
||||
currentGeneration: current.generation,
|
||||
currentObject: current.object,
|
||||
appliedBefore: runtime.flagsApplied,
|
||||
changes: request.changes,
|
||||
desired: object
|
||||
}
|
||||
// Before the generation check, so a dry run always shows the generation to pass next.
|
||||
log(JSON.stringify(plan))
|
||||
if (current.generation !== request.expectedGeneration) {
|
||||
throw generationConflict(current.generation, request.expectedGeneration)
|
||||
}
|
||||
// The cell applies every readable write; a generation it did not apply was voided there. A
|
||||
// write of an object this image accepts whole (checked above) repairs it.
|
||||
if (current.generation !== '0' && String(runtime.flagsApplied.generation) !== current.generation) {
|
||||
log(
|
||||
JSON.stringify({
|
||||
event: 'orca_relay_cell_flags_repairing_unapplied_object',
|
||||
cellId: request.cellId,
|
||||
appliedGeneration: runtime.flagsApplied.generation,
|
||||
currentGeneration: current.generation
|
||||
})
|
||||
)
|
||||
}
|
||||
const admitModeChange =
|
||||
request.changes.admitMode === undefined ? undefined : (request.changes.admitMode ?? 'db')
|
||||
if (admitModeChange !== undefined) {
|
||||
const recorded = await directorAdmitMode(fetchImpl, request, idToken, sleep)
|
||||
log(JSON.stringify({ event: 'orca_relay_cell_admit_mode_recorded_before', ...recorded }))
|
||||
}
|
||||
if (request.mode === 'dry-run') return { ...plan, written: false }
|
||||
// Sweeps skip the cell before it starts admitting from memory, never after.
|
||||
let generation
|
||||
if (admitModeChange === 'reserve') {
|
||||
const before = await directorAdmitMode(fetchImpl, request, idToken, sleep)
|
||||
await directorAdmitMode(fetchImpl, request, idToken, sleep, 'reserve')
|
||||
// Every director's cached set expires (5 s) before the cell starts admitting from memory;
|
||||
// the second write re-runs the open-flow check over anything started meanwhile.
|
||||
await sleep(RESERVE_MODE_CACHE_WAIT_MS)
|
||||
try {
|
||||
await directorAdmitMode(fetchImpl, request, idToken, sleep, 'reserve')
|
||||
} catch (error) {
|
||||
// A drain, migration or rehome opened during the wait: Postgres goes back as it was.
|
||||
if (before?.admitMode === 'db') await directorAdmitMode(fetchImpl, request, idToken, sleep, 'db')
|
||||
throw error
|
||||
}
|
||||
try {
|
||||
generation = await writeObject(fetchImpl, request, object, accessToken, audit)
|
||||
} catch (error) {
|
||||
// The file was refused outright (412, 4xx): put Postgres back as it was. After a 5xx the
|
||||
// object may be written, so Postgres keeps reserve (sweeps stay off: the safe side).
|
||||
if (error?.notWritten && before?.admitMode === 'db') {
|
||||
await directorAdmitMode(fetchImpl, request, idToken, sleep, 'db')
|
||||
}
|
||||
throw error
|
||||
}
|
||||
} else {
|
||||
generation = await writeObject(fetchImpl, request, object, accessToken, audit)
|
||||
}
|
||||
log(JSON.stringify({ event: 'orca_relay_cell_flags_written', cellId: request.cellId, generation }))
|
||||
const deadline = now() + READ_BACK_TIMEOUT_MS
|
||||
let last = runtime.flagsApplied
|
||||
while (now() < deadline) {
|
||||
await sleep(READ_BACK_INTERVAL_MS)
|
||||
try {
|
||||
last = (await readRuntime(fetchImpl, request, idToken, sleep)).flagsApplied
|
||||
} catch {
|
||||
// The object is written; a failed poll only delays the read-back until the deadline.
|
||||
continue
|
||||
}
|
||||
if (String(last?.generation) !== generation) continue
|
||||
// The image predates a switch this object names: it applied the rest and dropped that one.
|
||||
if (Array.isArray(last.ignoredKeys) && last.ignoredKeys.length > 0) {
|
||||
throw new Error(
|
||||
`cell applied generation ${generation} but ignores ${last.ignoredKeys.join(', ')}; ` +
|
||||
'its image predates them, so flip the object back'
|
||||
)
|
||||
}
|
||||
if (sameFlags(last.flags, object.flags)) {
|
||||
const result = { ...plan, written: true, generation, applied: last }
|
||||
log(JSON.stringify({ ...result, event: 'orca_relay_cell_flags_applied_read_back' }))
|
||||
if (admitModeChange === 'db') {
|
||||
await waitForLeasedControls(fetchImpl, request, idToken, { now, sleep, log })
|
||||
await directorAdmitMode(fetchImpl, request, idToken, sleep, 'db')
|
||||
log(JSON.stringify({ event: 'orca_relay_cell_admit_mode_recorded', cellId: request.cellId, admitMode: 'db' }))
|
||||
}
|
||||
return result
|
||||
}
|
||||
}
|
||||
throw new Error(
|
||||
`cell did not apply generation ${generation} within ${READ_BACK_TIMEOUT_MS} ms ` +
|
||||
`(last applied ${JSON.stringify(last)}); the object is written`
|
||||
)
|
||||
}
|
||||
|
||||
if (import.meta.url === pathToFileURL(process.argv[1] ?? '').href) {
|
||||
const { values } = parseArgs({
|
||||
options: Object.fromEntries(
|
||||
[
|
||||
'mode',
|
||||
'cell-id',
|
||||
'cell-origin',
|
||||
'project-id',
|
||||
'set',
|
||||
'expected-generation',
|
||||
'confirmation',
|
||||
'confirm-reserve',
|
||||
'break-glass',
|
||||
'director-origin'
|
||||
].map((name) => [name, { type: 'string' }])
|
||||
)
|
||||
})
|
||||
try {
|
||||
const request = parseCellFlagsRequest(values)
|
||||
const accessToken = process.env.ORCA_RELAY_CONTROL_ACCESS_TOKEN
|
||||
const idToken = process.env.ORCA_RELAY_ADMIN_ID_TOKEN
|
||||
if (!accessToken || !idToken) throw new Error('both tokens are required')
|
||||
await operateCellFlags(request, {
|
||||
accessToken,
|
||||
idToken,
|
||||
audit: {
|
||||
// The person who started this attempt, which differs from GITHUB_ACTOR on a re-run.
|
||||
writtenBy: process.env.GITHUB_TRIGGERING_ACTOR ?? process.env.GITHUB_ACTOR ?? 'unknown',
|
||||
runId: process.env.GITHUB_RUN_ID ?? 'local',
|
||||
runAttempt: process.env.GITHUB_RUN_ATTEMPT ?? '0'
|
||||
}
|
||||
})
|
||||
} catch (error) {
|
||||
// On stdout, so the job summary's copy of the run carries the reason.
|
||||
console.log(
|
||||
JSON.stringify({
|
||||
event: 'orca_relay_cell_flags_failed',
|
||||
message: error instanceof Error ? error.message : String(error)
|
||||
})
|
||||
)
|
||||
process.exitCode = 1
|
||||
}
|
||||
}
|
||||
@@ -1,611 +0,0 @@
|
||||
import assert from 'node:assert/strict'
|
||||
import { test } from 'node:test'
|
||||
import {
|
||||
assertSupportedObject,
|
||||
desiredObject,
|
||||
operateCellFlags,
|
||||
parseCellFlagsRequest,
|
||||
parseFlagChanges,
|
||||
READ_BACK_TIMEOUT_MS,
|
||||
REREGISTER_WAIT_MAX_MS,
|
||||
REREGISTER_WAIT_MIN_MS,
|
||||
reregisterLanes,
|
||||
reregisterWaitMs,
|
||||
WRITE_CONFIRMATION
|
||||
} from './operate-relay-cell-flags.mjs'
|
||||
|
||||
const CELL = 'production-gce-c26'
|
||||
const RESERVE = { 'director-origin': 'https://relay.onorca.dev', 'confirm-reserve': `RESERVE ${CELL}` }
|
||||
const values = (overrides = {}) => ({
|
||||
mode: 'write',
|
||||
'cell-id': CELL,
|
||||
'cell-origin': 'https://c26.relay.onorca.dev',
|
||||
'project-id': 'onorca-cloud',
|
||||
set: 'readinessLocal=true',
|
||||
'expected-generation': '0',
|
||||
confirmation: `${WRITE_CONFIRMATION} ${CELL}`,
|
||||
...overrides
|
||||
})
|
||||
|
||||
// A cell whose runtime status applies whatever was written, after `applyAfterPolls` polls.
|
||||
const SUPPORTED = {
|
||||
readinessLocal: { type: 'boolean' },
|
||||
ticketCheck: { type: 'enum', values: ['off', 'shadow', 'enforce'] },
|
||||
admitMode: { type: 'enum', values: ['db', 'reserve'] },
|
||||
intakePerSec: { type: 'number', min: 0, max: 1_000 },
|
||||
reserveDryRun: { type: 'boolean' },
|
||||
rejectionFence: { type: 'boolean' },
|
||||
readTimeoutMarginMs: { type: 'number', min: 1_000, max: 60_000, integer: true },
|
||||
reregisterInFlight: { type: 'number', min: 1, max: 16, integer: true },
|
||||
phonePath: { type: 'enum', values: ['db', 'memory'] }
|
||||
}
|
||||
|
||||
function fakeGoogleAndCell({
|
||||
stored = null,
|
||||
applyAfterPolls = 2,
|
||||
role = 'cell',
|
||||
cellId = CELL,
|
||||
supportedFlags = SUPPORTED,
|
||||
// Polls that still report admitModeEffective=reserve after a flip back.
|
||||
reregisteringPolls = 0,
|
||||
controls,
|
||||
// Forces admitModeEffective, as a tripped dead-man reports it.
|
||||
effectiveOverride,
|
||||
heartbeatFresh = false,
|
||||
// The director refuses this numbered reserve post (1-based) with this error.
|
||||
refuseReservePost
|
||||
} = {}) {
|
||||
const state = {
|
||||
stored,
|
||||
writes: [],
|
||||
polls: 0,
|
||||
// A cell that has applied whatever object is already there.
|
||||
applied: stored
|
||||
? { generation: Number(stored.generation), flags: stored.object.flags }
|
||||
: { generation: 0, flags: { readinessLocal: false, ticketCheck: 'off' } },
|
||||
directorAdmitMode: 'db',
|
||||
reservePosts: 0,
|
||||
order: [],
|
||||
reregisteringPolls
|
||||
}
|
||||
const fetchImpl = async (url, init = {}) => {
|
||||
const target = new URL(url)
|
||||
if (target.pathname === '/v1/admin/cell-admit-mode') {
|
||||
const body = JSON.parse(String(init.body))
|
||||
if (body.admitMode === 'reserve') {
|
||||
state.reservePosts += 1
|
||||
if (refuseReservePost?.post === state.reservePosts) {
|
||||
return Response.json({ error: refuseReservePost.error }, { status: 409 })
|
||||
}
|
||||
}
|
||||
if (body.admitMode) {
|
||||
state.directorAdmitMode = body.admitMode
|
||||
state.order.push(`director:${body.admitMode}`)
|
||||
}
|
||||
return Response.json({ v: 1, cellId: body.cellId, admitMode: state.directorAdmitMode, updatedAt: 1, heartbeatFresh })
|
||||
}
|
||||
if (target.pathname === '/v1/admin/runtime-status') {
|
||||
state.polls += 1
|
||||
if (state.stored && state.polls > applyAfterPolls) {
|
||||
state.applied = { generation: Number(state.stored.generation), flags: state.stored.object.flags }
|
||||
}
|
||||
const reserveApplied = state.applied.flags.admitMode === 'reserve'
|
||||
const effective = reserveApplied || (state.writes.length > 0 && state.reregisteringPolls-- > 0) ? 'reserve' : 'db'
|
||||
return Response.json({
|
||||
v: 1,
|
||||
role,
|
||||
cellId,
|
||||
flagsApplied: state.applied,
|
||||
admitModeEffective: effectiveOverride ?? effective,
|
||||
...(controls === undefined ? {} : { runtime: { controls } }),
|
||||
...(supportedFlags ? { supportedFlags } : {})
|
||||
})
|
||||
}
|
||||
if (target.pathname.startsWith('/upload/')) {
|
||||
const expected = target.searchParams.get('ifGenerationMatch')
|
||||
if (expected !== (state.stored?.generation ?? '0')) return new Response(null, { status: 412 })
|
||||
const parts = String(init.body).split('\r\n')
|
||||
const object = JSON.parse(parts[parts.indexOf('Content-Type: application/json') + 2])
|
||||
const metadata = JSON.parse(parts[3])
|
||||
const generation = String(Number(state.stored?.generation ?? 1000) + 1)
|
||||
state.stored = { generation, object }
|
||||
state.writes.push({ generation, object, metadata })
|
||||
state.order.push(`file:${object.flags.admitMode ?? 'db'}`)
|
||||
return Response.json({ generation })
|
||||
}
|
||||
if (!state.stored) return new Response(null, { status: 404 })
|
||||
if (target.searchParams.get('alt') === 'media') return Response.json(state.stored.object)
|
||||
return Response.json({ generation: state.stored.generation })
|
||||
}
|
||||
return { state, fetchImpl }
|
||||
}
|
||||
|
||||
const run = (request, fake, extra = {}) => {
|
||||
let clock = 0
|
||||
const lines = []
|
||||
return {
|
||||
lines,
|
||||
result: operateCellFlags(request, {
|
||||
fetchImpl: fake.fetchImpl,
|
||||
accessToken: 'access',
|
||||
idToken: 'id',
|
||||
audit: { writtenBy: 'octocat', runId: '42', runAttempt: '1' },
|
||||
log: (line) => lines.push(JSON.parse(line)),
|
||||
now: () => clock,
|
||||
sleep: async (ms) => {
|
||||
clock += ms
|
||||
},
|
||||
...extra
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
test('refuses malformed requests before touching anything', () => {
|
||||
assert.throws(() => parseCellFlagsRequest(values({ mode: 'apply' })), /mode/)
|
||||
assert.throws(() => parseCellFlagsRequest(values({ 'cell-id': 'all' })), /cell id/)
|
||||
assert.throws(() => parseCellFlagsRequest(values({ set: 'ticketCheck=always' })), /ticketCheck/)
|
||||
assert.throws(() => parseCellFlagsRequest(values({ set: 'placerMode=on' })), /unknown switch/)
|
||||
assert.throws(() => parseCellFlagsRequest(values({ set: '' })), /names no switch/)
|
||||
assert.throws(() => parseCellFlagsRequest(values({ set: 'admitMode=db,admitMode=reserve' })), /twice/)
|
||||
assert.throws(() => parseCellFlagsRequest(values({ set: 'readTimeoutMarginMs=500' })), /cannot be/)
|
||||
assert.throws(() => parseCellFlagsRequest(values({ 'expected-generation': '' })), /expected/)
|
||||
assert.throws(() => parseCellFlagsRequest(values({ confirmation: WRITE_CONFIRMATION })), /confirmation/)
|
||||
assert.throws(
|
||||
() => parseCellFlagsRequest(values({ 'cell-origin': 'https://c27.relay.onorca.dev' })),
|
||||
/another cell/
|
||||
)
|
||||
assert.throws(() => parseCellFlagsRequest(values({ 'cell-origin': 'https://evil.example' })), /origin/)
|
||||
// A dry run needs no confirmation.
|
||||
assert.equal(parseCellFlagsRequest(values({ mode: 'dry-run', confirmation: '' })).mode, 'dry-run')
|
||||
})
|
||||
|
||||
test('a dry run reports the change and writes nothing', async () => {
|
||||
const fake = fakeGoogleAndCell()
|
||||
const { result, lines } = run(parseCellFlagsRequest(values({ mode: 'dry-run' })), fake)
|
||||
const outcome = await result
|
||||
assert.equal(outcome.written, false)
|
||||
assert.equal(fake.state.writes.length, 0)
|
||||
assert.equal(lines[0].event, 'orca_relay_cell_flags_plan')
|
||||
assert.deepEqual(lines[0].desired.flags, { readinessLocal: true })
|
||||
})
|
||||
|
||||
test('writes with compare-and-swap, records who wrote it, and waits for the cell to apply it', async () => {
|
||||
const fake = fakeGoogleAndCell()
|
||||
const { result, lines } = run(parseCellFlagsRequest(values()), fake)
|
||||
const outcome = await result
|
||||
assert.equal(outcome.written, true)
|
||||
assert.equal(outcome.generation, '1001')
|
||||
assert.deepEqual(outcome.applied, { generation: 1001, flags: { readinessLocal: true } })
|
||||
assert.deepEqual(fake.state.writes[0].metadata.metadata, { writtenBy: 'octocat', runId: '42', runAttempt: '1' })
|
||||
assert.deepEqual(fake.state.writes[0].object, {
|
||||
v: 1,
|
||||
cellId: CELL,
|
||||
flags: { readinessLocal: true }
|
||||
})
|
||||
assert.deepEqual(lines.map((line) => line.event), [
|
||||
'orca_relay_cell_flags_plan',
|
||||
'orca_relay_cell_flags_written',
|
||||
'orca_relay_cell_flags_applied_read_back'
|
||||
])
|
||||
// Flip back: the next write must name the generation just written.
|
||||
const back = run(
|
||||
parseCellFlagsRequest(values({ set: 'readinessLocal=false', 'expected-generation': '1001' })),
|
||||
fake
|
||||
)
|
||||
assert.equal((await back.result).generation, '1002')
|
||||
})
|
||||
|
||||
test('refuses a stale expected generation, and a concurrent writer between read and write', async () => {
|
||||
const fake = fakeGoogleAndCell({
|
||||
stored: { generation: '7', object: { v: 1, cellId: CELL, flags: {} } }
|
||||
})
|
||||
await assert.rejects(run(parseCellFlagsRequest(values()), fake).result, /current generation is 7/)
|
||||
assert.equal(fake.state.writes.length, 0)
|
||||
const racing = fakeGoogleAndCell()
|
||||
const write = racing.fetchImpl
|
||||
racing.fetchImpl = async (url, init) => {
|
||||
if (new URL(url).pathname.startsWith('/upload/')) racing.state.stored = { generation: '9', object: {} }
|
||||
return await write(url, init)
|
||||
}
|
||||
await assert.rejects(run(parseCellFlagsRequest(values()), racing).result, /changed since/)
|
||||
})
|
||||
|
||||
test('a dry run against a stale generation still shows the plan, then refuses', async () => {
|
||||
const fake = fakeGoogleAndCell({
|
||||
stored: { generation: '7', object: { v: 1, cellId: CELL, flags: {} } }
|
||||
})
|
||||
const { result, lines } = run(parseCellFlagsRequest(values({ mode: 'dry-run' })), fake)
|
||||
await assert.rejects(result, /current generation is 7/)
|
||||
assert.equal(lines[0].event, 'orca_relay_cell_flags_plan')
|
||||
assert.equal(lines[0].currentGeneration, '7')
|
||||
})
|
||||
|
||||
test('rides out a transient 5xx on the first read and during the read-back', async () => {
|
||||
const fake = fakeGoogleAndCell()
|
||||
const inner = fake.fetchImpl
|
||||
let runtimeCalls = 0
|
||||
fake.fetchImpl = async (url, init) => {
|
||||
if (new URL(url).pathname === '/v1/admin/runtime-status') {
|
||||
runtimeCalls += 1
|
||||
// The first call, and the first read-back poll after the write.
|
||||
if (runtimeCalls === 1 || runtimeCalls === 3) return new Response(null, { status: 503 })
|
||||
}
|
||||
return await inner(url, init)
|
||||
}
|
||||
const outcome = await run(parseCellFlagsRequest(values()), fake).result
|
||||
assert.equal(outcome.written, true)
|
||||
assert.equal(outcome.generation, '1001')
|
||||
})
|
||||
|
||||
test('refuses an origin that is not the named cell, or an image without the flag channel', async () => {
|
||||
await assert.rejects(
|
||||
run(parseCellFlagsRequest(values()), fakeGoogleAndCell({ cellId: 'production-gce-c25' })).result,
|
||||
/not production-gce-c26/
|
||||
)
|
||||
const old = fakeGoogleAndCell()
|
||||
const fetchImpl = old.fetchImpl
|
||||
old.fetchImpl = async (url, init) =>
|
||||
new URL(url).pathname === '/v1/admin/runtime-status'
|
||||
? Response.json({ v: 1, role: 'cell', cellId: CELL })
|
||||
: await fetchImpl(url, init)
|
||||
await assert.rejects(run(parseCellFlagsRequest(values()), old).result, /no flag channel/)
|
||||
})
|
||||
|
||||
test('fails loudly when the cell does not apply the write within the read-back window', async () => {
|
||||
const fake = fakeGoogleAndCell({ applyAfterPolls: READ_BACK_TIMEOUT_MS })
|
||||
await assert.rejects(run(parseCellFlagsRequest(values()), fake).result, /did not apply generation 1001/)
|
||||
assert.equal(fake.state.writes.length, 1)
|
||||
})
|
||||
|
||||
test('changes only the named switches and keeps every other key the object already has', () => {
|
||||
const current = {
|
||||
v: 1,
|
||||
cellId: CELL,
|
||||
flags: { readinessLocal: true, ticketCheck: 'shadow', readTimeoutMarginMs: 5_000 }
|
||||
}
|
||||
const request = parseCellFlagsRequest(
|
||||
values({ set: 'admitMode=reserve, intakePerSec=40, readTimeoutMarginMs=default', ...RESERVE })
|
||||
)
|
||||
assert.deepEqual(desiredObject(request, current), {
|
||||
v: 1,
|
||||
cellId: CELL,
|
||||
flags: { readinessLocal: true, ticketCheck: 'shadow', admitMode: 'reserve', intakePerSec: 40 }
|
||||
})
|
||||
assert.deepEqual(desiredObject(request, null).flags, { admitMode: 'reserve', intakePerSec: 40 })
|
||||
assert.throws(() => desiredObject(request, { v: 1, cellId: 'production-gce-c27', flags: {} }), /by hand/)
|
||||
assert.deepEqual(parseFlagChanges('rejectionFence=false'), { rejectionFence: false })
|
||||
assert.deepEqual(parseFlagChanges('reregisterInFlight=8'), { reregisterInFlight: 8 })
|
||||
assert.throws(() => parseFlagChanges('reregisterInFlight=32'), /cannot be/)
|
||||
})
|
||||
|
||||
test('a write keeps the switches it did not name, end to end', async () => {
|
||||
const fake = fakeGoogleAndCell({
|
||||
stored: { generation: '7', object: { v: 1, cellId: CELL, flags: { ticketCheck: 'shadow' } } }
|
||||
})
|
||||
const outcome = await run(parseCellFlagsRequest(values({ 'expected-generation': '7' })), fake).result
|
||||
assert.equal(outcome.written, true)
|
||||
assert.deepEqual(fake.state.writes[0].object.flags, { ticketCheck: 'shadow', readinessLocal: true })
|
||||
})
|
||||
|
||||
test('the read-back compares every switch, not only the one this run changed', async () => {
|
||||
const fake = fakeGoogleAndCell({
|
||||
stored: { generation: '7', object: { v: 1, cellId: CELL, flags: { ticketCheck: 'shadow' } } }
|
||||
})
|
||||
const inner = fake.fetchImpl
|
||||
fake.fetchImpl = async (url, init) => {
|
||||
const response = await inner(url, init)
|
||||
if (new URL(url).pathname !== '/v1/admin/runtime-status') return response
|
||||
const body = await response.json()
|
||||
// The cell reports the new generation but still runs the old ticketCheck.
|
||||
if (body.flagsApplied.generation === 8) body.flagsApplied.flags = { readinessLocal: true }
|
||||
return Response.json(body)
|
||||
}
|
||||
await assert.rejects(
|
||||
run(parseCellFlagsRequest(values({ 'expected-generation': '7' })), fake).result,
|
||||
/did not apply generation 8/
|
||||
)
|
||||
})
|
||||
|
||||
test('a switch the cell reports on its own is compared only when the object names it', async () => {
|
||||
const fake = fakeGoogleAndCell()
|
||||
const inner = fake.fetchImpl
|
||||
fake.fetchImpl = async (url, init) => {
|
||||
const response = await inner(url, init)
|
||||
if (new URL(url).pathname !== '/v1/admin/runtime-status') return response
|
||||
const body = await response.json()
|
||||
body.flagsApplied.flags = { ...body.flagsApplied.flags, intakePerSec: 50, readTimeoutMarginMs: 10_000 }
|
||||
return Response.json(body)
|
||||
}
|
||||
assert.equal((await run(parseCellFlagsRequest(values()), fake).result).written, true)
|
||||
})
|
||||
|
||||
test('fails the read-back when the cell ignores a switch the object names', async () => {
|
||||
const fake = fakeGoogleAndCell()
|
||||
const inner = fake.fetchImpl
|
||||
fake.fetchImpl = async (url, init) => {
|
||||
const response = await inner(url, init)
|
||||
if (new URL(url).pathname !== '/v1/admin/runtime-status') return response
|
||||
const body = await response.json()
|
||||
if (body.flagsApplied.generation === 1001) body.flagsApplied.ignoredKeys = ['admitMode']
|
||||
return Response.json(body)
|
||||
}
|
||||
await assert.rejects(
|
||||
run(parseCellFlagsRequest(values({ set: 'readinessLocal=true,admitMode=reserve', ...RESERVE })), fake).result,
|
||||
/ignores admitMode/
|
||||
)
|
||||
})
|
||||
|
||||
test('refuses a switch the cell image does not support, before writing anything', async () => {
|
||||
const old = fakeGoogleAndCell({ supportedFlags: null })
|
||||
await assert.rejects(
|
||||
run(parseCellFlagsRequest(values({ set: 'ticketCheck=enforce' })), old).result,
|
||||
/does not support ticketCheck="enforce"/
|
||||
)
|
||||
await assert.rejects(
|
||||
run(parseCellFlagsRequest(values({ set: 'admitMode=reserve', ...RESERVE })), old).result,
|
||||
/does not support admitMode/
|
||||
)
|
||||
assert.equal(old.state.writes.length, 0)
|
||||
// An image from before supportedFlags still takes the two switches it always had.
|
||||
assert.equal((await run(parseCellFlagsRequest(values()), old).result).written, true)
|
||||
assert.throws(
|
||||
() => assertSupportedObject({ readTimeoutMarginMs: 1_500.5 }, { supportedFlags: SUPPORTED }),
|
||||
/readTimeoutMarginMs/
|
||||
)
|
||||
assertSupportedObject({ ticketCheck: 'enforce' }, { supportedFlags: SUPPORTED })
|
||||
})
|
||||
|
||||
test('refuses to carry a bad value already in the file', async () => {
|
||||
const bad = fakeGoogleAndCell({
|
||||
supportedFlags: null,
|
||||
stored: { generation: '7', object: { v: 1, cellId: CELL, flags: { ticketCheck: 'enforce' } } }
|
||||
})
|
||||
// The old image voided generation 7 and still runs generation 0.
|
||||
bad.state.applied = { generation: 0, flags: { readinessLocal: false, ticketCheck: 'off' } }
|
||||
await assert.rejects(
|
||||
run(parseCellFlagsRequest(values({ 'expected-generation': '7' })), bad).result,
|
||||
/does not support ticketCheck="enforce"/
|
||||
)
|
||||
assert.equal(bad.state.writes.length, 0)
|
||||
})
|
||||
|
||||
const DIRECTOR = { 'director-origin': RESERVE['director-origin'] }
|
||||
|
||||
test('admitMode=reserve needs a typed confirmation, and Postgres records reserve before the file', async () => {
|
||||
assert.throws(
|
||||
() => parseCellFlagsRequest(values({ ...DIRECTOR, set: 'admitMode=reserve' })),
|
||||
/--confirm-reserve "RESERVE production-gce-c26"/
|
||||
)
|
||||
assert.throws(() => parseCellFlagsRequest(values({ set: 'admitMode=db' })), /--director-origin/)
|
||||
const fake = fakeGoogleAndCell()
|
||||
const request = parseCellFlagsRequest(
|
||||
values({ ...DIRECTOR, set: 'readinessLocal=true,admitMode=reserve', 'confirm-reserve': `RESERVE ${CELL}` })
|
||||
)
|
||||
assert.equal((await run(request, fake).result).written, true)
|
||||
// Recorded, re-checked after every director's cache has expired, then the file.
|
||||
assert.deepEqual(fake.state.order, ['director:reserve', 'director:reserve', 'file:reserve'])
|
||||
})
|
||||
|
||||
test('a refused file write after Postgres says reserve puts Postgres back to db', async () => {
|
||||
const fake = fakeGoogleAndCell()
|
||||
const inner = fake.fetchImpl
|
||||
fake.fetchImpl = async (url, init) => {
|
||||
if (new URL(url).pathname.startsWith('/upload/')) return new Response(null, { status: 403 })
|
||||
return await inner(url, init)
|
||||
}
|
||||
const request = parseCellFlagsRequest(values({ set: 'readinessLocal=true,admitMode=reserve', ...RESERVE }))
|
||||
await assert.rejects(run(request, fake).result, /returned 403/)
|
||||
assert.deepEqual(fake.state.order, ['director:reserve', 'director:reserve', 'director:db'])
|
||||
})
|
||||
|
||||
test('repairs an object the cell did not apply when the whole new object is valid', async () => {
|
||||
const voided = fakeGoogleAndCell({
|
||||
stored: { generation: '7', object: { v: 1, cellId: CELL, flags: { readinessLocal: true } } }
|
||||
})
|
||||
voided.state.applied = { generation: 3, flags: { readinessLocal: false } }
|
||||
const { result, lines } = run(parseCellFlagsRequest(values({ 'expected-generation': '7' })), voided)
|
||||
assert.equal((await result).written, true)
|
||||
assert.ok(lines.some((line) => line.event === 'orca_relay_cell_flags_repairing_unapplied_object'))
|
||||
})
|
||||
|
||||
test('a flip back records db in Postgres only after the cell has leased every control', async () => {
|
||||
const fake = fakeGoogleAndCell({
|
||||
stored: { generation: '7', object: { v: 1, cellId: CELL, flags: { admitMode: 'reserve' } } },
|
||||
reregisteringPolls: 3
|
||||
})
|
||||
fake.state.directorAdmitMode = 'reserve'
|
||||
const request = parseCellFlagsRequest(values({ ...DIRECTOR, set: 'admitMode=default', 'expected-generation': '7' }))
|
||||
assert.equal((await run(request, fake).result).written, true)
|
||||
assert.deepEqual(fake.state.order, ['file:db', 'director:db'])
|
||||
assert.equal(fake.state.reregisteringPolls, -1)
|
||||
})
|
||||
|
||||
test('the flip-back wait scales with the control count and the switch-file pace', () => {
|
||||
assert.equal(reregisterWaitMs({ runtime: { controls: 100 } }), REREGISTER_WAIT_MIN_MS)
|
||||
// 1,500 controls at the default 3 lanes: 1,667 s.
|
||||
assert.equal(reregisterWaitMs({ runtime: { controls: 1_500 } }), 1_666_667)
|
||||
assert.equal(
|
||||
reregisterWaitMs({ runtime: { controls: 1_500 }, flagsApplied: { flags: { reregisterInFlight: 10 } } }),
|
||||
// The cell caps 10 at its US pool of 10 less two.
|
||||
625_000
|
||||
)
|
||||
assert.equal(reregisterWaitMs({ runtime: { controls: 3_000 } }), REREGISTER_WAIT_MAX_MS)
|
||||
})
|
||||
|
||||
test('the wait counts the lanes the cell actually uses', () => {
|
||||
assert.equal(reregisterLanes({ region: 'us-central1' }), 3)
|
||||
assert.equal(reregisterLanes({ region: 'asia-east2' }), 5)
|
||||
assert.equal(reregisterLanes({ region: 'us-central1', flagsApplied: { flags: { reregisterInFlight: 16 } } }), 8)
|
||||
assert.equal(reregisterLanes({ region: 'asia-east2', flagsApplied: { flags: { reregisterInFlight: 16 } } }), 14)
|
||||
// The pool the cell reports wins over the deployed-size table.
|
||||
assert.equal(reregisterLanes({ region: 'us-central1', databasePoolMax: 22 }), 7)
|
||||
assert.equal(reregisterLanes({ region: 'asia-east2', databasePoolMax: 12, flagsApplied: { flags: { reregisterInFlight: 16 } } }), 10)
|
||||
assert.equal(reregisterLanes({ region: 'asia-east2', databasePoolMax: 'x' }), 5)
|
||||
})
|
||||
|
||||
test('a tripped cell takes no write that keeps reserve, only one that also says db', async () => {
|
||||
const stored = { generation: '7', object: { v: 1, cellId: CELL, flags: { admitMode: 'reserve' } } }
|
||||
const tripped = () => fakeGoogleAndCell({ stored, effectiveOverride: 'db' })
|
||||
for (const set of ['readinessLocal=true', 'reregisterInFlight=4']) {
|
||||
const fake = tripped()
|
||||
const request = parseCellFlagsRequest(values({ set, 'expected-generation': '7' }))
|
||||
await assert.rejects(run(request, fake).result, /tripped its dead-man[\s\S]*admitMode=db,reregisterInFlight=N/)
|
||||
assert.deepEqual(fake.state.writes, [])
|
||||
}
|
||||
const fake = tripped()
|
||||
fake.state.directorAdmitMode = 'reserve'
|
||||
const request = parseCellFlagsRequest(
|
||||
values({ ...DIRECTOR, set: 'admitMode=db,reregisterInFlight=4', 'expected-generation': '7' })
|
||||
)
|
||||
assert.equal((await run(request, fake).result).written, true)
|
||||
assert.deepEqual(fake.state.writes[0].object.flags, { admitMode: 'db', reregisterInFlight: 4 })
|
||||
})
|
||||
|
||||
test('reserve requires readinessLocal=true, already written or in the same write', async () => {
|
||||
const refused = fakeGoogleAndCell()
|
||||
await assert.rejects(
|
||||
run(parseCellFlagsRequest(values({ set: 'admitMode=reserve', ...RESERVE })), refused).result,
|
||||
/admitMode=reserve requires readinessLocal=true/
|
||||
)
|
||||
assert.deepEqual(refused.state.order, [])
|
||||
assert.deepEqual(refused.state.writes, [])
|
||||
|
||||
// Already written by an earlier run: reserve alone is enough.
|
||||
const stored = { generation: '7', object: { v: 1, cellId: CELL, flags: { readinessLocal: true } } }
|
||||
const ready = fakeGoogleAndCell({ stored })
|
||||
const request = parseCellFlagsRequest(
|
||||
values({ set: 'admitMode=reserve', 'expected-generation': '7', ...RESERVE })
|
||||
)
|
||||
assert.equal((await run(request, ready).result).written, true)
|
||||
assert.deepEqual(ready.state.writes[0].object.flags, { readinessLocal: true, admitMode: 'reserve' })
|
||||
})
|
||||
|
||||
test('a reserve cell refuses readinessLocal=false, but takes it together with the flip back', async () => {
|
||||
const stored = {
|
||||
generation: '7',
|
||||
object: { v: 1, cellId: CELL, flags: { readinessLocal: true, admitMode: 'reserve' } }
|
||||
}
|
||||
for (const set of ['readinessLocal=false', 'readinessLocal=default']) {
|
||||
const fake = fakeGoogleAndCell({ stored })
|
||||
const request = parseCellFlagsRequest(values({ set, 'expected-generation': '7' }))
|
||||
await assert.rejects(run(request, fake).result, /readinessLocal must stay true while the cell is reserve/)
|
||||
assert.deepEqual(fake.state.writes, [])
|
||||
}
|
||||
const fake = fakeGoogleAndCell({ stored })
|
||||
fake.state.directorAdmitMode = 'reserve'
|
||||
const request = parseCellFlagsRequest(
|
||||
values({ ...DIRECTOR, set: 'admitMode=db,readinessLocal=false', 'expected-generation': '7' })
|
||||
)
|
||||
assert.equal((await run(request, fake).result).written, true)
|
||||
assert.deepEqual(fake.state.writes[0].object.flags, { readinessLocal: false, admitMode: 'db' })
|
||||
})
|
||||
|
||||
test('phonePath=memory needs reserve, and a reserve flip never turns phones over unnamed', async () => {
|
||||
const onDatabase = { generation: '7', object: { v: 1, cellId: CELL, flags: { readinessLocal: true } } }
|
||||
const refused = fakeGoogleAndCell({ stored: onDatabase })
|
||||
await assert.rejects(
|
||||
run(parseCellFlagsRequest(values({ set: 'phonePath=memory', 'expected-generation': '7' })), refused).result,
|
||||
/phonePath=memory requires admitMode=reserve/
|
||||
)
|
||||
assert.deepEqual(refused.state.writes, [])
|
||||
|
||||
const together = fakeGoogleAndCell({ stored: onDatabase })
|
||||
const both = parseCellFlagsRequest(
|
||||
values({ set: 'admitMode=reserve,phonePath=memory', 'expected-generation': '7', ...RESERVE })
|
||||
)
|
||||
assert.equal((await run(both, together).result).written, true)
|
||||
assert.deepEqual(together.state.writes[0].object.flags, {
|
||||
readinessLocal: true,
|
||||
admitMode: 'reserve',
|
||||
phonePath: 'memory'
|
||||
})
|
||||
|
||||
const reserve = { readinessLocal: true, admitMode: 'reserve' }
|
||||
const onReserve = fakeGoogleAndCell({ stored: { ...onDatabase, object: { ...onDatabase.object, flags: reserve } } })
|
||||
const alone = parseCellFlagsRequest(values({ set: 'phonePath=memory', 'expected-generation': '7' }))
|
||||
assert.equal((await run(alone, onReserve).result).written, true)
|
||||
|
||||
// Rolling back with admitMode=db leaves phonePath=memory in the object, which the cell ignores.
|
||||
const memory = { ...reserve, phonePath: 'memory' }
|
||||
const rollback = fakeGoogleAndCell({ stored: { ...onDatabase, object: { ...onDatabase.object, flags: memory } } })
|
||||
rollback.state.directorAdmitMode = 'reserve'
|
||||
const toDatabase = parseCellFlagsRequest(values({ ...DIRECTOR, set: 'admitMode=db', 'expected-generation': '7' }))
|
||||
assert.equal((await run(toDatabase, rollback).result).written, true)
|
||||
assert.deepEqual(rollback.state.writes[0].object.flags, { ...memory, admitMode: 'db' })
|
||||
|
||||
// phonePath=db removes the key, so an image without the switch still takes later writes.
|
||||
const off = fakeGoogleAndCell({ stored: { ...onDatabase, object: { ...onDatabase.object, flags: memory } } })
|
||||
assert.equal((await run(parseCellFlagsRequest(values({ set: 'phonePath=db', 'expected-generation': '7' })), off).result).written, true)
|
||||
assert.deepEqual(off.state.writes[0].object.flags, reserve)
|
||||
|
||||
const lingering = { readinessLocal: true, admitMode: 'db', phonePath: 'memory' }
|
||||
const unnamed = fakeGoogleAndCell({ stored: { ...onDatabase, object: { ...onDatabase.object, flags: lingering } } })
|
||||
await assert.rejects(
|
||||
run(parseCellFlagsRequest(values({ set: 'admitMode=reserve', 'expected-generation': '7', ...RESERVE })), unnamed)
|
||||
.result,
|
||||
/still says phonePath=memory/
|
||||
)
|
||||
assert.deepEqual(unnamed.state.writes, [])
|
||||
})
|
||||
|
||||
test('a migration opened during the cache wait puts Postgres back to db', async () => {
|
||||
const fake = fakeGoogleAndCell({ refuseReservePost: { post: 2, error: 'cell_has_open_migration' } })
|
||||
const request = parseCellFlagsRequest(values({ ...RESERVE, set: 'readinessLocal=true,admitMode=reserve' }))
|
||||
await assert.rejects(run(request, fake).result, /cell_has_open_migration/)
|
||||
assert.deepEqual(fake.state.order, ['director:reserve', 'director:db'])
|
||||
assert.deepEqual(fake.state.writes, [])
|
||||
})
|
||||
|
||||
test('break glass refuses a cell that answers or that the director still hears', async () => {
|
||||
const glass = { ...RESERVE, set: 'admitMode=default', 'expected-generation': '7', 'break-glass': `BREAK GLASS ${CELL}` }
|
||||
const stored = { generation: '7', object: { v: 1, cellId: CELL, flags: { admitMode: 'reserve' } } }
|
||||
const answering = fakeGoogleAndCell({ stored })
|
||||
await assert.rejects(run(parseCellFlagsRequest(values(glass)), answering).result, /answers runtime-status/)
|
||||
const heard = fakeGoogleAndCell({ stored, heartbeatFresh: true })
|
||||
const inner = heard.fetchImpl
|
||||
heard.fetchImpl = async (url, init) => {
|
||||
if (new URL(url).pathname === '/v1/admin/runtime-status') throw new Error('connect refused')
|
||||
return await inner(url, init)
|
||||
}
|
||||
await assert.rejects(run(parseCellFlagsRequest(values(glass)), heard).result, /still sees heartbeats/)
|
||||
assert.deepEqual(heard.state.writes, [])
|
||||
assert.deepEqual(answering.state.writes, [])
|
||||
})
|
||||
|
||||
test('a flip back that outlasts its wait keeps Postgres on reserve and says how to finish', async () => {
|
||||
const fake = fakeGoogleAndCell({
|
||||
stored: { generation: '7', object: { v: 1, cellId: CELL, flags: { admitMode: 'reserve' } } },
|
||||
reregisteringPolls: Number.POSITIVE_INFINITY,
|
||||
controls: 1_500
|
||||
})
|
||||
fake.state.directorAdmitMode = 'reserve'
|
||||
const request = parseCellFlagsRequest(values({ ...DIRECTOR, set: 'admitMode=default', 'expected-generation': '7' }))
|
||||
const { lines, result } = run(request, fake)
|
||||
await assert.rejects(result, /after 1666667 ms[\s\S]*Run again with --set admitMode=db[\s\S]*--set admitMode=db,reregisterInFlight=N/)
|
||||
assert.deepEqual(fake.state.order, ['file:db'])
|
||||
assert.equal(lines.find((line) => line.event === 'orca_relay_cell_reregistration_wait')?.waitMs, 1_666_667)
|
||||
})
|
||||
|
||||
test('break glass records db (and drops phonePath) for a cell that cannot answer, typed for that cell only', async () => {
|
||||
const glass = { ...RESERVE, set: 'admitMode=default', 'expected-generation': '7' }
|
||||
assert.throws(() => parseCellFlagsRequest(values({ ...glass, 'break-glass': 'BREAK GLASS production-gce-c27' })), /break glass requires/)
|
||||
assert.throws(
|
||||
() => parseCellFlagsRequest(values({ ...glass, set: 'admitMode=default,readinessLocal=true', 'break-glass': `BREAK GLASS ${CELL}` })),
|
||||
/on its own/
|
||||
)
|
||||
const fake = fakeGoogleAndCell({
|
||||
stored: { generation: '7', object: { v: 1, cellId: CELL, flags: { admitMode: 'reserve', readinessLocal: true, phonePath: 'memory' } } }
|
||||
})
|
||||
fake.state.directorAdmitMode = 'reserve'
|
||||
const inner = fake.fetchImpl
|
||||
fake.fetchImpl = async (url, init) => {
|
||||
if (new URL(url).pathname === '/v1/admin/runtime-status') throw new Error('connect refused')
|
||||
return await inner(url, init)
|
||||
}
|
||||
const { result, lines } = run(parseCellFlagsRequest(values({ ...glass, 'break-glass': `BREAK GLASS ${CELL}` })), fake)
|
||||
assert.equal((await result).written, true)
|
||||
assert.deepEqual(fake.state.writes[0].object.flags, { readinessLocal: true })
|
||||
assert.deepEqual(fake.state.order, ['file:db', 'director:db'])
|
||||
assert.equal(lines.at(-1).event, 'orca_relay_cell_admit_mode_break_glass')
|
||||
})
|
||||
|
||||
@@ -1,344 +0,0 @@
|
||||
import { pathToFileURL } from 'node:url'
|
||||
import { fetchAdminOnceMore } from './relay-admin-transient-retry.mjs'
|
||||
import { inspectAdmissionSelector } from './relay-admission-selector.mjs'
|
||||
|
||||
const DIRECTOR_ORIGIN = 'https://relay.onorca.dev'
|
||||
const MODES = new Set(['inspect', 'enable', 'pause', 'disable', 'recover-enable'])
|
||||
// c1-c99 with no leading zero; a fixed ceiling here once refused a newly general cell.
|
||||
const PRODUCTION_CELL_ID = /^production-gce-c[1-9][0-9]?$/
|
||||
|
||||
function canonicalCells(value) {
|
||||
if (value === 'none') return []
|
||||
const cells = value.split(',').map((cell) => cell.trim()).filter(Boolean).sort()
|
||||
if (
|
||||
cells.length === 0 ||
|
||||
new Set(cells).size !== cells.length ||
|
||||
cells.some((cell) => !PRODUCTION_CELL_ID.test(cell))
|
||||
) throw new Error('selector membership is invalid')
|
||||
return cells
|
||||
}
|
||||
|
||||
function integer(value, name, { minimum = 0, maximum = Number.MAX_SAFE_INTEGER } = {}) {
|
||||
const parsed = Number(value)
|
||||
if (!Number.isSafeInteger(parsed) || parsed < minimum || parsed > maximum) {
|
||||
throw new Error(`${name} is invalid`)
|
||||
}
|
||||
return parsed
|
||||
}
|
||||
|
||||
export function parseRegionalRehomeArguments(argv, environment = process.env) {
|
||||
const values = {}
|
||||
for (let index = 0; index < argv.length; index += 2) {
|
||||
const key = argv[index]
|
||||
const value = argv[index + 1]
|
||||
if (!key?.startsWith('--') || value === undefined) throw new Error('invalid arguments')
|
||||
values[key.slice(2)] = value
|
||||
}
|
||||
for (const key of ['mode', 'director-origin', 'expected-control-generation']) {
|
||||
if (!values[key]) throw new Error(`missing --${key}`)
|
||||
}
|
||||
if (!MODES.has(values.mode)) throw new Error('--mode is invalid')
|
||||
if (values['director-origin'] !== DIRECTOR_ORIGIN) {
|
||||
throw new Error('--director-origin must be the production Relay origin')
|
||||
}
|
||||
const recovery = values.mode === 'recover-enable'
|
||||
const mutation = values.mode !== 'inspect' && !recovery
|
||||
const mutationKeys = [
|
||||
'not-before',
|
||||
'rate-per-minute',
|
||||
'preference-max-age-ms',
|
||||
'host-cooldown-ms',
|
||||
'drain-grace-ms',
|
||||
'confirmation'
|
||||
]
|
||||
if (mutation && mutationKeys.some((key) => values[key] === undefined)) {
|
||||
throw new Error('mutations require the complete durable control shape')
|
||||
}
|
||||
if (!mutation && !recovery && mutationKeys.some((key) => values[key] !== undefined)) {
|
||||
throw new Error('inspect cannot carry mutation arguments')
|
||||
}
|
||||
const selectorKeys = [
|
||||
'expected-selector-generation',
|
||||
'expected-existing-only-cells',
|
||||
'expected-migration-only-cells',
|
||||
'expected-general-cells'
|
||||
]
|
||||
if (!recovery && selectorKeys.some((key) => values[key] === undefined)) {
|
||||
throw new Error('operation requires exact selector state')
|
||||
}
|
||||
if (recovery && selectorKeys.some((key) => values[key] !== undefined)) {
|
||||
throw new Error('enable recovery cannot depend on selector diagnostics')
|
||||
}
|
||||
const expectedConfirmation = {
|
||||
enable: 'ENABLE_REGIONAL_REHOMING',
|
||||
pause: 'PAUSE_REGIONAL_REHOMING',
|
||||
disable: 'DISABLE_REGIONAL_REHOMING'
|
||||
}[values.mode]
|
||||
if (mutation && values.confirmation !== expectedConfirmation) {
|
||||
throw new Error('confirmation does not match the requested control action')
|
||||
}
|
||||
if (recovery && values.confirmation !== 'RECOVER_FAILED_REGIONAL_REHOME_ENABLE') {
|
||||
throw new Error('confirmation does not authorize failed-enable recovery')
|
||||
}
|
||||
if (
|
||||
recovery &&
|
||||
mutationKeys
|
||||
.filter((key) => key !== 'confirmation')
|
||||
.some((key) => values[key] !== undefined)
|
||||
) throw new Error('enable recovery cannot carry durable control shape arguments')
|
||||
const token = environment.ORCA_RELAY_ADMIN_ID_TOKEN
|
||||
if (!token || token.length > 8_192) throw new Error('admin identity token is unavailable')
|
||||
return {
|
||||
mode: values.mode,
|
||||
directorOrigin: DIRECTOR_ORIGIN,
|
||||
...(!recovery
|
||||
? {
|
||||
expectedSelectorGeneration: integer(
|
||||
values['expected-selector-generation'],
|
||||
'--expected-selector-generation'
|
||||
),
|
||||
expectedMembership: {
|
||||
existingOnly: canonicalCells(values['expected-existing-only-cells']),
|
||||
migrationOnly: canonicalCells(values['expected-migration-only-cells']),
|
||||
general: canonicalCells(values['expected-general-cells'])
|
||||
}
|
||||
}
|
||||
: {}),
|
||||
expectedControlGeneration: integer(
|
||||
values['expected-control-generation'],
|
||||
'--expected-control-generation'
|
||||
),
|
||||
...(mutation
|
||||
? {
|
||||
notBefore: integer(values['not-before'], '--not-before'),
|
||||
ratePerMinute: integer(values['rate-per-minute'], '--rate-per-minute', {
|
||||
minimum: 1,
|
||||
maximum: 120
|
||||
}),
|
||||
preferenceMaxAgeMs: integer(
|
||||
values['preference-max-age-ms'],
|
||||
'--preference-max-age-ms',
|
||||
{ minimum: 60_000, maximum: 30 * 24 * 60 * 60_000 }
|
||||
),
|
||||
hostCooldownMs: integer(
|
||||
values['host-cooldown-ms'],
|
||||
'--host-cooldown-ms',
|
||||
{ minimum: 60_000, maximum: 30 * 24 * 60 * 60_000 }
|
||||
),
|
||||
drainGraceMs: integer(values['drain-grace-ms'], '--drain-grace-ms', {
|
||||
minimum: 60_000,
|
||||
maximum: 60 * 60_000
|
||||
})
|
||||
}
|
||||
: {}),
|
||||
token
|
||||
}
|
||||
}
|
||||
|
||||
async function responseJson(response, label) {
|
||||
const body = await response.json().catch(() => ({}))
|
||||
if (!response.ok) throw new Error(`${label} returned ${response.status}: ${body.error ?? 'unknown'}`)
|
||||
return body
|
||||
}
|
||||
|
||||
function exactMembership(actual, expected) {
|
||||
return ['existingOnly', 'migrationOnly', 'general'].every(
|
||||
(key) => JSON.stringify(actual[key]) === JSON.stringify(expected[key])
|
||||
)
|
||||
}
|
||||
|
||||
function assertControl(control, expected) {
|
||||
if (
|
||||
(expected.generation !== undefined && control?.generation !== expected.generation) ||
|
||||
typeof control.enabled !== 'boolean' ||
|
||||
!Number.isSafeInteger(control.observationStartedAt) ||
|
||||
!Number.isSafeInteger(control.notBefore) ||
|
||||
!Number.isSafeInteger(control.ratePerMinute) ||
|
||||
!Number.isSafeInteger(control.preferenceMaxAgeMs) ||
|
||||
// A director predating the per-host cooldown does not report it. Reading
|
||||
// the control and both emergency brakes must keep working against that
|
||||
// image; only enable requires the field.
|
||||
(control.hostCooldownMs !== undefined &&
|
||||
!Number.isSafeInteger(control.hostCooldownMs)) ||
|
||||
!Number.isSafeInteger(control.drainGraceMs)
|
||||
) throw new Error('director returned an invalid regional rehome control')
|
||||
if (expected.enabled !== undefined && control.enabled !== expected.enabled) {
|
||||
throw new Error('regional rehome enabled state does not match')
|
||||
}
|
||||
return control
|
||||
}
|
||||
|
||||
// Echo the cooldown only when the director already reports it: a legacy
|
||||
// director rejects the unknown key outright and would refuse every brake.
|
||||
function cooldownField(before, value) {
|
||||
return before.hostCooldownMs === undefined ? {} : { hostCooldownMs: value }
|
||||
}
|
||||
|
||||
async function verifiedDisabledControl(post, generation) {
|
||||
return assertControl((await post('/v1/admin/regional-rehome-control', {
|
||||
v: 1,
|
||||
action: 'inspect'
|
||||
})).control, { generation, enabled: false })
|
||||
}
|
||||
|
||||
async function applyDisabledControl(post, before) {
|
||||
return assertControl((await post('/v1/admin/regional-rehome-control', {
|
||||
v: 1,
|
||||
action: 'apply',
|
||||
expectedGeneration: before.generation,
|
||||
enabled: false,
|
||||
notBefore: before.notBefore,
|
||||
ratePerMinute: before.ratePerMinute,
|
||||
preferenceMaxAgeMs: before.preferenceMaxAgeMs,
|
||||
...cooldownField(before, before.hostCooldownMs),
|
||||
drainGraceMs: before.drainGraceMs,
|
||||
confirmation: 'DISABLE_REGIONAL_REHOMING'
|
||||
})).control, { generation: before.generation + 1, enabled: false })
|
||||
}
|
||||
|
||||
async function resolveAmbiguousDisable(post, before, firstError) {
|
||||
const observed = assertControl((await post('/v1/admin/regional-rehome-control', {
|
||||
v: 1,
|
||||
action: 'inspect'
|
||||
})).control, {})
|
||||
if (observed.generation === before.generation + 1 && !observed.enabled) {
|
||||
return observed
|
||||
}
|
||||
if (observed.generation !== before.generation || !observed.enabled) {
|
||||
throw new AggregateError(
|
||||
[firstError],
|
||||
'failed-enable recovery reached an unexpected control generation'
|
||||
)
|
||||
}
|
||||
try {
|
||||
return await applyDisabledControl(post, before)
|
||||
} catch (retryError) {
|
||||
try {
|
||||
return await verifiedDisabledControl(post, before.generation + 1)
|
||||
} catch (readbackError) {
|
||||
throw new AggregateError(
|
||||
[firstError, retryError, readbackError],
|
||||
'failed-enable recovery exhausted two bounded CAS attempts'
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export async function recoverRegionalRehomeEnable(config, post) {
|
||||
const before = assertControl((await post('/v1/admin/regional-rehome-control', {
|
||||
v: 1,
|
||||
action: 'inspect'
|
||||
})).control, {})
|
||||
if (
|
||||
before.generation < config.expectedControlGeneration ||
|
||||
(before.generation === config.expectedControlGeneration && before.enabled)
|
||||
) throw new Error('durable control cannot belong to the failed enable attempt')
|
||||
if (!before.enabled) {
|
||||
const verified = await verifiedDisabledControl(post, before.generation)
|
||||
return { mode: config.mode, recovered: false, control: verified }
|
||||
}
|
||||
let applied
|
||||
try {
|
||||
applied = await applyDisabledControl(post, before)
|
||||
} catch (error) {
|
||||
applied = await resolveAmbiguousDisable(post, before, error)
|
||||
}
|
||||
const verified = await verifiedDisabledControl(post, applied.generation)
|
||||
return { mode: config.mode, recovered: true, control: verified }
|
||||
}
|
||||
|
||||
export async function operateRegionalRehome(config, dependencies = {}) {
|
||||
const fetchImpl = dependencies.fetch ?? fetch
|
||||
const post = dependencies.post ?? (async (path, body) => await responseJson(
|
||||
// Generation-guarded writes make a retry a no-op or an explicit mismatch, never a double apply.
|
||||
await fetchAdminOnceMore(
|
||||
fetchImpl,
|
||||
`${config.directorOrigin}${path}`,
|
||||
{
|
||||
method: 'POST',
|
||||
headers: {
|
||||
authorization: `Bearer ${config.token}`,
|
||||
'content-type': 'application/json'
|
||||
},
|
||||
body: JSON.stringify(body)
|
||||
},
|
||||
{ wait: dependencies.wait }
|
||||
),
|
||||
path
|
||||
))
|
||||
if (config.mode === 'recover-enable') {
|
||||
return await recoverRegionalRehomeEnable(config, post)
|
||||
}
|
||||
const selector = (await inspectAdmissionSelector(post)).selector
|
||||
if (
|
||||
selector.generation !== config.expectedSelectorGeneration ||
|
||||
!exactMembership(selector.membership, config.expectedMembership)
|
||||
) throw new Error('admission selector does not match the reviewed generation and membership')
|
||||
|
||||
const inspected = await post('/v1/admin/regional-rehome-control', {
|
||||
v: 1,
|
||||
action: 'inspect'
|
||||
})
|
||||
const before = assertControl(inspected.control, {
|
||||
generation: config.expectedControlGeneration
|
||||
})
|
||||
if (config.mode === 'inspect') return { mode: config.mode, selector, control: before }
|
||||
if (config.mode === 'enable' && before.enabled) {
|
||||
throw new Error('regional rehome is already enabled; inspect before changing its rate')
|
||||
}
|
||||
if (config.mode === 'pause' && !before.enabled) {
|
||||
throw new Error('regional rehome is already paused')
|
||||
}
|
||||
const enabled = config.mode === 'enable'
|
||||
if (enabled && before.hostCooldownMs === undefined) {
|
||||
throw new Error(
|
||||
'director does not report a per-host rehome cooldown; deploy a director that supports it before enabling'
|
||||
)
|
||||
}
|
||||
const applied = await post('/v1/admin/regional-rehome-control', {
|
||||
v: 1,
|
||||
action: 'apply',
|
||||
expectedGeneration: config.expectedControlGeneration,
|
||||
enabled,
|
||||
notBefore: config.notBefore,
|
||||
ratePerMinute: config.ratePerMinute,
|
||||
preferenceMaxAgeMs: config.preferenceMaxAgeMs,
|
||||
...cooldownField(before, config.hostCooldownMs),
|
||||
drainGraceMs: config.drainGraceMs,
|
||||
confirmation: enabled
|
||||
? 'ENABLE_REGIONAL_REHOMING'
|
||||
: 'DISABLE_REGIONAL_REHOMING'
|
||||
})
|
||||
const after = assertControl(applied.control, {
|
||||
generation: config.expectedControlGeneration + 1,
|
||||
enabled
|
||||
})
|
||||
const verified = assertControl((await post('/v1/admin/regional-rehome-control', {
|
||||
v: 1,
|
||||
action: 'inspect'
|
||||
})).control, {
|
||||
generation: after.generation,
|
||||
enabled
|
||||
})
|
||||
return { mode: config.mode, selector, control: verified }
|
||||
}
|
||||
|
||||
export async function main(
|
||||
argv = process.argv.slice(2),
|
||||
environment = process.env,
|
||||
dependencies = {},
|
||||
write = (value) => process.stdout.write(value)
|
||||
) {
|
||||
const result = await operateRegionalRehome(
|
||||
parseRegionalRehomeArguments(argv, environment),
|
||||
dependencies
|
||||
)
|
||||
write(`${JSON.stringify({ event: 'relay_regional_rehome_control', ...result })}\n`)
|
||||
}
|
||||
|
||||
if (import.meta.url === pathToFileURL(process.argv[1]).href) {
|
||||
main().catch((error) => {
|
||||
process.stderr.write(`${error instanceof Error ? error.message : String(error)}\n`)
|
||||
process.exitCode = 1
|
||||
})
|
||||
}
|
||||
@@ -1,446 +0,0 @@
|
||||
import assert from 'node:assert/strict'
|
||||
import { test } from 'node:test'
|
||||
import {
|
||||
main,
|
||||
operateRegionalRehome,
|
||||
parseRegionalRehomeArguments,
|
||||
recoverRegionalRehomeEnable
|
||||
} from './operate-relay-regional-rehome.mjs'
|
||||
|
||||
const membership = {
|
||||
existingOnly: ['production-gce-c1'],
|
||||
migrationOnly: ['production-gce-c2'],
|
||||
general: ['production-gce-c7', 'production-gce-c27']
|
||||
}
|
||||
|
||||
function argumentsFor(mode, confirmation) {
|
||||
return [
|
||||
'--mode', mode,
|
||||
'--director-origin', 'https://relay.onorca.dev',
|
||||
'--expected-selector-generation', '11',
|
||||
'--expected-existing-only-cells', membership.existingOnly.join(','),
|
||||
'--expected-migration-only-cells', membership.migrationOnly.join(','),
|
||||
'--expected-general-cells', membership.general.join(','),
|
||||
'--expected-control-generation', '4',
|
||||
...(mode === 'inspect' ? [] : [
|
||||
'--not-before', '2000000000000',
|
||||
'--rate-per-minute', '10',
|
||||
'--preference-max-age-ms', '86400000',
|
||||
'--host-cooldown-ms', '604800000',
|
||||
'--drain-grace-ms', '60000',
|
||||
'--confirmation', confirmation
|
||||
])
|
||||
]
|
||||
}
|
||||
|
||||
function control(generation, enabled) {
|
||||
return {
|
||||
generation,
|
||||
enabled,
|
||||
observationStartedAt: 1,
|
||||
notBefore: 2_000_000_000_000,
|
||||
ratePerMinute: 10,
|
||||
preferenceMaxAgeMs: 86_400_000,
|
||||
hostCooldownMs: 604_800_000,
|
||||
drainGraceMs: 60_000
|
||||
}
|
||||
}
|
||||
|
||||
// The control a director predating the per-host cooldown reports.
|
||||
function legacyControl(generation, enabled) {
|
||||
const { hostCooldownMs: _absent, ...rest } = control(generation, enabled)
|
||||
return rest
|
||||
}
|
||||
|
||||
function legacyDirector(controls) {
|
||||
const requests = []
|
||||
const post = async (path, body) => {
|
||||
requests.push({ path, body })
|
||||
if (path === '/v1/admin/admission-selector/status') {
|
||||
return { selector: { generation: 11, membership } }
|
||||
}
|
||||
return { v: 1, control: controls.shift() }
|
||||
}
|
||||
return { requests, post }
|
||||
}
|
||||
|
||||
test('parses exact selector and typed control confirmation', () => {
|
||||
const parsed = parseRegionalRehomeArguments(
|
||||
argumentsFor('enable', 'ENABLE_REGIONAL_REHOMING'),
|
||||
{ ORCA_RELAY_ADMIN_ID_TOKEN: 'token' }
|
||||
)
|
||||
assert.equal(parsed.expectedSelectorGeneration, 11)
|
||||
assert.equal(parsed.expectedControlGeneration, 4)
|
||||
assert.equal(parsed.ratePerMinute, 10)
|
||||
assert.equal(parsed.hostCooldownMs, 604_800_000)
|
||||
assert.throws(
|
||||
() => parseRegionalRehomeArguments(
|
||||
argumentsFor('enable', 'ENABLE_REGIONAL_REHOMING').filter(
|
||||
(value, index, all) =>
|
||||
value !== '--host-cooldown-ms' && all[index - 1] !== '--host-cooldown-ms'
|
||||
),
|
||||
{ ORCA_RELAY_ADMIN_ID_TOKEN: 'token' }
|
||||
),
|
||||
/complete durable control shape/
|
||||
)
|
||||
assert.throws(
|
||||
() => parseRegionalRehomeArguments(
|
||||
argumentsFor('pause', 'DISABLE_REGIONAL_REHOMING'),
|
||||
{ ORCA_RELAY_ADMIN_ID_TOKEN: 'token' }
|
||||
),
|
||||
/confirmation/
|
||||
)
|
||||
assert.throws(
|
||||
() => parseRegionalRehomeArguments(
|
||||
argumentsFor('inspect').concat('--rate-per-minute', '10'),
|
||||
{ ORCA_RELAY_ADMIN_ID_TOKEN: 'token' }
|
||||
),
|
||||
/inspect cannot/
|
||||
)
|
||||
})
|
||||
|
||||
test('accepts every one- and two-digit production cell and rejects malformed membership', () => {
|
||||
const parseGeneral = (general) => parseRegionalRehomeArguments(
|
||||
argumentsFor('inspect').map((value, index, all) =>
|
||||
all[index - 1] === '--expected-general-cells' ? general : value
|
||||
),
|
||||
{ ORCA_RELAY_ADMIN_ID_TOKEN: 'token' }
|
||||
).expectedMembership.general
|
||||
assert.deepEqual(
|
||||
parseGeneral('production-gce-c30,production-gce-c9,production-gce-c29'),
|
||||
['production-gce-c29', 'production-gce-c30', 'production-gce-c9']
|
||||
)
|
||||
assert.deepEqual(parseGeneral('production-gce-c99'), ['production-gce-c99'])
|
||||
for (const general of [
|
||||
'production-gce-c0',
|
||||
'production-gce-c100',
|
||||
'production-gce-c09',
|
||||
'production-gce-c30,production-gce-c30',
|
||||
'staging-gce-c30',
|
||||
','
|
||||
]) {
|
||||
assert.throws(() => parseGeneral(general), /selector membership is invalid/, general)
|
||||
}
|
||||
})
|
||||
|
||||
test('binds enable to exact selector and durable control generations', async () => {
|
||||
const requests = []
|
||||
const controls = [
|
||||
{ generation: 4, enabled: false },
|
||||
{ generation: 5, enabled: true },
|
||||
{ generation: 5, enabled: true }
|
||||
].map((control) => ({
|
||||
observationStartedAt: 1,
|
||||
notBefore: 0,
|
||||
ratePerMinute: 10,
|
||||
preferenceMaxAgeMs: 86_400_000,
|
||||
hostCooldownMs: 604_800_000,
|
||||
drainGraceMs: 60_000,
|
||||
...control
|
||||
}))
|
||||
const config = parseRegionalRehomeArguments(
|
||||
argumentsFor('enable', 'ENABLE_REGIONAL_REHOMING'),
|
||||
{ ORCA_RELAY_ADMIN_ID_TOKEN: 'token' }
|
||||
)
|
||||
const result = await operateRegionalRehome(config, {
|
||||
post: async (path, body) => {
|
||||
requests.push({ path, body })
|
||||
if (path === '/v1/admin/admission-selector/status') {
|
||||
return { selector: { generation: 11, membership } }
|
||||
}
|
||||
return { v: 1, control: controls.shift() }
|
||||
}
|
||||
})
|
||||
assert.equal(result.control.generation, 5)
|
||||
assert.equal(result.control.hostCooldownMs, 604_800_000)
|
||||
assert.deepEqual(requests[2].body, {
|
||||
v: 1,
|
||||
action: 'apply',
|
||||
expectedGeneration: 4,
|
||||
enabled: true,
|
||||
notBefore: 2_000_000_000_000,
|
||||
ratePerMinute: 10,
|
||||
preferenceMaxAgeMs: 86_400_000,
|
||||
hostCooldownMs: 604_800_000,
|
||||
drainGraceMs: 60_000,
|
||||
confirmation: 'ENABLE_REGIONAL_REHOMING'
|
||||
})
|
||||
})
|
||||
|
||||
test('inspects a director that predates the per-host cooldown', async () => {
|
||||
const director = legacyDirector([legacyControl(4, true)])
|
||||
const config = parseRegionalRehomeArguments(
|
||||
argumentsFor('inspect'),
|
||||
{ ORCA_RELAY_ADMIN_ID_TOKEN: 'token' }
|
||||
)
|
||||
|
||||
const result = await operateRegionalRehome(config, { post: director.post })
|
||||
|
||||
assert.equal(result.control.generation, 4)
|
||||
assert.equal(result.control.hostCooldownMs, undefined)
|
||||
})
|
||||
|
||||
for (const [mode, confirmation, enabledBefore] of [
|
||||
['pause', 'PAUSE_REGIONAL_REHOMING', true],
|
||||
['disable', 'DISABLE_REGIONAL_REHOMING', false]
|
||||
]) {
|
||||
test(`${mode} still brakes a director that predates the cooldown`, async () => {
|
||||
const director = legacyDirector([
|
||||
legacyControl(4, enabledBefore),
|
||||
legacyControl(5, false),
|
||||
legacyControl(5, false)
|
||||
])
|
||||
const config = parseRegionalRehomeArguments(
|
||||
argumentsFor(mode, confirmation),
|
||||
{ ORCA_RELAY_ADMIN_ID_TOKEN: 'token' }
|
||||
)
|
||||
|
||||
const result = await operateRegionalRehome(config, { post: director.post })
|
||||
|
||||
assert.equal(result.control.generation, 5)
|
||||
// The unknown key would be refused by that director's strict schema.
|
||||
assert.equal('hostCooldownMs' in director.requests[2].body, false)
|
||||
assert.equal(director.requests[2].body.confirmation, 'DISABLE_REGIONAL_REHOMING')
|
||||
})
|
||||
}
|
||||
|
||||
test('failed-enable recovery brakes a director that predates the cooldown', async () => {
|
||||
const requests = []
|
||||
let current = legacyControl(7, true)
|
||||
const result = await recoverRegionalRehomeEnable({
|
||||
mode: 'recover-enable',
|
||||
expectedControlGeneration: 4
|
||||
}, async (_path, body) => {
|
||||
requests.push(body)
|
||||
if (body.action === 'inspect') return { control: current }
|
||||
current = legacyControl(8, false)
|
||||
return { control: current }
|
||||
})
|
||||
|
||||
assert.equal(result.control.generation, 8)
|
||||
assert.equal('hostCooldownMs' in requests[1], false)
|
||||
})
|
||||
|
||||
test('refuses to enable a director that does not report the cooldown', async () => {
|
||||
const director = legacyDirector([legacyControl(4, false)])
|
||||
const config = parseRegionalRehomeArguments(
|
||||
argumentsFor('enable', 'ENABLE_REGIONAL_REHOMING'),
|
||||
{ ORCA_RELAY_ADMIN_ID_TOKEN: 'token' }
|
||||
)
|
||||
|
||||
await assert.rejects(
|
||||
operateRegionalRehome(config, { post: director.post }),
|
||||
/per-host rehome cooldown/
|
||||
)
|
||||
// Read-only: selector status and the control inspect, and nothing else.
|
||||
assert.equal(director.requests.length, 2)
|
||||
assert.equal(director.requests.every(({ body }) => body.action !== 'apply'), true)
|
||||
})
|
||||
|
||||
test('fails closed on selector drift before reading or mutating control', async () => {
|
||||
let calls = 0
|
||||
const config = parseRegionalRehomeArguments(
|
||||
argumentsFor('disable', 'DISABLE_REGIONAL_REHOMING'),
|
||||
{ ORCA_RELAY_ADMIN_ID_TOKEN: 'token' }
|
||||
)
|
||||
await assert.rejects(
|
||||
operateRegionalRehome(config, {
|
||||
post: async () => {
|
||||
calls += 1
|
||||
return { selector: { generation: 12, membership } }
|
||||
}
|
||||
}),
|
||||
/selector/
|
||||
)
|
||||
assert.equal(calls, 1)
|
||||
})
|
||||
|
||||
test('failed-enable recovery CAS-disables an advanced enabled generation', async () => {
|
||||
const requests = []
|
||||
let current = control(7, true)
|
||||
const result = await recoverRegionalRehomeEnable({
|
||||
mode: 'recover-enable',
|
||||
expectedControlGeneration: 4
|
||||
}, async (_path, body) => {
|
||||
requests.push(body)
|
||||
if (body.action === 'inspect') return { control: current }
|
||||
assert.equal(body.expectedGeneration, 7)
|
||||
current = control(8, false)
|
||||
throw new Error('enable recovery response was lost')
|
||||
})
|
||||
assert.equal(result.recovered, true)
|
||||
assert.deepEqual(result.control, control(8, false))
|
||||
assert.deepEqual(requests[1], {
|
||||
v: 1,
|
||||
action: 'apply',
|
||||
expectedGeneration: 7,
|
||||
enabled: false,
|
||||
notBefore: 2_000_000_000_000,
|
||||
ratePerMinute: 10,
|
||||
preferenceMaxAgeMs: 86_400_000,
|
||||
hostCooldownMs: 604_800_000,
|
||||
drainGraceMs: 60_000,
|
||||
confirmation: 'DISABLE_REGIONAL_REHOMING'
|
||||
})
|
||||
})
|
||||
|
||||
test('failed-enable recovery retries once when the first CAS never commits', async () => {
|
||||
let current = control(7, true)
|
||||
const applyRequests = []
|
||||
const result = await recoverRegionalRehomeEnable({
|
||||
mode: 'recover-enable',
|
||||
expectedControlGeneration: 4
|
||||
}, async (_path, body) => {
|
||||
if (body.action === 'inspect') return { control: current }
|
||||
applyRequests.push(body)
|
||||
if (applyRequests.length === 1) {
|
||||
throw new Error('disable request was lost before commit')
|
||||
}
|
||||
current = control(8, false)
|
||||
throw new Error('retry response was lost after commit')
|
||||
})
|
||||
assert.equal(applyRequests.length, 2)
|
||||
assert.deepEqual(applyRequests[1], applyRequests[0])
|
||||
assert.equal(result.recovered, true)
|
||||
assert.deepEqual(result.control, control(8, false))
|
||||
})
|
||||
|
||||
test('failed-enable recovery stops after two uncommitted CAS attempts', async () => {
|
||||
let applyCalls = 0
|
||||
await assert.rejects(
|
||||
recoverRegionalRehomeEnable({
|
||||
mode: 'recover-enable',
|
||||
expectedControlGeneration: 4
|
||||
}, async (_path, body) => {
|
||||
if (body.action === 'inspect') return { control: control(7, true) }
|
||||
applyCalls += 1
|
||||
throw new Error(`disable attempt ${applyCalls} was lost before commit`)
|
||||
}),
|
||||
/exhausted two bounded CAS attempts/
|
||||
)
|
||||
assert.equal(applyCalls, 2)
|
||||
})
|
||||
|
||||
test('failed-enable recovery is a verified no-op before enable and after cleanup', async () => {
|
||||
for (const current of [control(4, false), control(8, false)]) {
|
||||
const requests = []
|
||||
const result = await recoverRegionalRehomeEnable({
|
||||
mode: 'recover-enable',
|
||||
expectedControlGeneration: 4
|
||||
}, async (_path, body) => {
|
||||
requests.push(body)
|
||||
return { control: current }
|
||||
})
|
||||
assert.equal(result.recovered, false)
|
||||
assert.equal(result.control.enabled, false)
|
||||
assert.deepEqual(requests.map(({ action }) => action), ['inspect', 'inspect'])
|
||||
}
|
||||
})
|
||||
|
||||
test('failed-enable recovery rejects an unchanged pre-existing enabled state', async () => {
|
||||
await assert.rejects(
|
||||
recoverRegionalRehomeEnable({
|
||||
mode: 'recover-enable',
|
||||
expectedControlGeneration: 4
|
||||
}, async () => ({ control: control(4, true) })),
|
||||
/cannot belong to the failed enable attempt/
|
||||
)
|
||||
})
|
||||
|
||||
test('parses recovery without depending on selector diagnostics', () => {
|
||||
const recoveryArguments = [
|
||||
'--mode', 'recover-enable',
|
||||
'--director-origin', 'https://relay.onorca.dev',
|
||||
'--expected-control-generation', '4',
|
||||
'--confirmation', 'RECOVER_FAILED_REGIONAL_REHOME_ENABLE'
|
||||
]
|
||||
const parsed = parseRegionalRehomeArguments(
|
||||
recoveryArguments,
|
||||
{ ORCA_RELAY_ADMIN_ID_TOKEN: 'token' }
|
||||
)
|
||||
assert.equal(parsed.expectedControlGeneration, 4)
|
||||
assert.equal(parsed.expectedMembership, undefined)
|
||||
assert.throws(
|
||||
() => parseRegionalRehomeArguments(
|
||||
recoveryArguments.concat('--not-before', '2000000000000'),
|
||||
{ ORCA_RELAY_ADMIN_ID_TOKEN: 'token' }
|
||||
),
|
||||
/cannot carry durable control shape/
|
||||
)
|
||||
})
|
||||
|
||||
test('main executes recovery mode and emits verified disabled control', async () => {
|
||||
let current = control(5, true)
|
||||
let output = ''
|
||||
await main([
|
||||
'--mode', 'recover-enable',
|
||||
'--director-origin', 'https://relay.onorca.dev',
|
||||
'--expected-control-generation', '4',
|
||||
'--confirmation', 'RECOVER_FAILED_REGIONAL_REHOME_ENABLE'
|
||||
], { ORCA_RELAY_ADMIN_ID_TOKEN: 'token' }, {
|
||||
post: async (_path, body) => {
|
||||
if (body.action === 'apply') current = control(6, false)
|
||||
return { control: current }
|
||||
}
|
||||
}, (value) => {
|
||||
output += value
|
||||
})
|
||||
assert.deepEqual(JSON.parse(output), {
|
||||
event: 'relay_regional_rehome_control',
|
||||
mode: 'recover-enable',
|
||||
recovered: true,
|
||||
control: control(6, false)
|
||||
})
|
||||
})
|
||||
|
||||
test('retries a transient 503 on the director control endpoint', async () => {
|
||||
const config = parseRegionalRehomeArguments(
|
||||
argumentsFor('inspect'),
|
||||
{ ORCA_RELAY_ADMIN_ID_TOKEN: 'token' }
|
||||
)
|
||||
const paths = []
|
||||
let selectorCalls = 0
|
||||
const result = await operateRegionalRehome(config, {
|
||||
wait: async () => {},
|
||||
fetch: async (url) => {
|
||||
const path = new URL(url).pathname
|
||||
paths.push(path)
|
||||
if (path === '/v1/admin/admission-selector/status') {
|
||||
selectorCalls += 1
|
||||
// The first read of each admin path 503s the way a warming instance does.
|
||||
if (selectorCalls === 1) return new Response('warming up', { status: 503 })
|
||||
return Response.json({ selector: { generation: 11, membership } })
|
||||
}
|
||||
if (paths.filter((value) => value === path).length === 1) {
|
||||
return new Response('warming up', { status: 503 })
|
||||
}
|
||||
return Response.json({ v: 1, control: control(4, false) })
|
||||
}
|
||||
})
|
||||
assert.equal(result.control.generation, 4)
|
||||
assert.deepEqual(paths, [
|
||||
'/v1/admin/admission-selector/status',
|
||||
'/v1/admin/admission-selector/status',
|
||||
'/v1/admin/regional-rehome-control',
|
||||
'/v1/admin/regional-rehome-control'
|
||||
])
|
||||
})
|
||||
|
||||
test('fails when both attempts at the director control endpoint return 503', async () => {
|
||||
const config = parseRegionalRehomeArguments(
|
||||
argumentsFor('inspect'),
|
||||
{ ORCA_RELAY_ADMIN_ID_TOKEN: 'token' }
|
||||
)
|
||||
let calls = 0
|
||||
await assert.rejects(
|
||||
operateRegionalRehome(config, {
|
||||
wait: async () => {},
|
||||
fetch: async () => {
|
||||
calls += 1
|
||||
return new Response('warming up', { status: 503 })
|
||||
}
|
||||
}),
|
||||
/returned 503/
|
||||
)
|
||||
assert.equal(calls, 2)
|
||||
})
|
||||
@@ -1,537 +0,0 @@
|
||||
import { readFileSync } from 'node:fs'
|
||||
import { spawnSync } from 'node:child_process'
|
||||
import { pathToFileURL } from 'node:url'
|
||||
import {
|
||||
inspectAdmissionSelector,
|
||||
selectorCellState
|
||||
} from './relay-admission-selector.mjs'
|
||||
|
||||
const PROJECT = 'onorca-cloud-staging'
|
||||
const REGION = 'us-central1'
|
||||
const DIRECTOR_ORIGIN = 'https://relay-staging.onorca.dev'
|
||||
const ADMIN_AUDIENCE = `${DIRECTOR_ORIGIN}/v1/admin/drain`
|
||||
const SQL_INSTANCE = 'orca-cloud-staging-auth-db'
|
||||
const CLOUD_RUN_SERVICES = [
|
||||
{ name: 'orca-cloud-relay-staging', healthOrigin: DIRECTOR_ORIGIN },
|
||||
{ name: 'orca-cloud-auth-staging', healthOrigin: 'https://auth-staging.onorca.dev' }
|
||||
]
|
||||
const POLL_INTERVAL_MS = 5_000
|
||||
const WAKE_TIMEOUT_MS = 12 * 60 * 1_000
|
||||
|
||||
export function parseArguments(argv) {
|
||||
const values = {}
|
||||
for (let index = 0; index < argv.length; index += 2) {
|
||||
const key = argv[index]
|
||||
const value = argv[index + 1]
|
||||
if (!key?.startsWith('--') || value === undefined) throw new Error(`invalid argument ${key ?? ''}`)
|
||||
const name = key.slice(2)
|
||||
if (!['mode', 'wake-cells', 'topology-file'].includes(name)) {
|
||||
throw new Error(`unsupported argument --${name}`)
|
||||
}
|
||||
values[name] = value
|
||||
}
|
||||
if (!['status', 'sleep', 'wake'].includes(values.mode)) {
|
||||
throw new Error('--mode must be status, sleep, or wake')
|
||||
}
|
||||
if (!['configured', 'all'].includes(values['wake-cells'])) {
|
||||
throw new Error('--wake-cells must be configured or all')
|
||||
}
|
||||
if (!values['topology-file']) throw new Error('missing --topology-file')
|
||||
return {
|
||||
mode: values.mode,
|
||||
wakeCells: values['wake-cells'],
|
||||
topologyFile: values['topology-file']
|
||||
}
|
||||
}
|
||||
|
||||
function canonicalStagingCell(cellId, value) {
|
||||
if (!/^staging-gce-[a-z0-9-]+$/.test(cellId)) throw new Error(`unsafe staging cell ID ${cellId}`)
|
||||
if (!value || typeof value !== 'object') throw new Error(`missing topology for ${cellId}`)
|
||||
const cell = {
|
||||
cellId,
|
||||
migName: String(value.mig_name ?? ''),
|
||||
zone: String(value.zone ?? ''),
|
||||
origin: String(value.origin ?? ''),
|
||||
initiallyEnabled: value.initially_enabled
|
||||
}
|
||||
if (!/^orca-cloud-staging-relay-gce-[a-z0-9-]+$/.test(cell.migName)) {
|
||||
throw new Error(`${cellId} has an unsafe MIG name`)
|
||||
}
|
||||
if (!/^(?:us-central1|asia-east2)-[a-z]$/.test(cell.zone)) {
|
||||
throw new Error(`${cellId} has an unsafe zone`)
|
||||
}
|
||||
const origin = new URL(cell.origin)
|
||||
if (
|
||||
origin.protocol !== 'https:' ||
|
||||
origin.origin !== cell.origin ||
|
||||
!origin.hostname.endsWith('.relay-staging.onorca.dev')
|
||||
) {
|
||||
throw new Error(`${cellId} has an unsafe origin`)
|
||||
}
|
||||
if (typeof cell.initiallyEnabled !== 'boolean') {
|
||||
throw new Error(`${cellId} has no initial admission state`)
|
||||
}
|
||||
return cell
|
||||
}
|
||||
|
||||
export function readStagingTopology(file) {
|
||||
const parsed = JSON.parse(readFileSync(file, 'utf8'))
|
||||
if (!parsed || typeof parsed !== 'object' || Array.isArray(parsed)) {
|
||||
throw new Error('staging topology must be an object')
|
||||
}
|
||||
const cells = Object.entries(parsed)
|
||||
.map(([cellId, value]) => canonicalStagingCell(cellId, value))
|
||||
.sort((left, right) => left.cellId.localeCompare(right.cellId))
|
||||
if (cells.length < 2 || cells.length > 10) {
|
||||
throw new Error('staging topology must contain 2..10 cells')
|
||||
}
|
||||
if (cells.filter((cell) => cell.initiallyEnabled).length < 2) {
|
||||
throw new Error('staging topology must retain two configured admission cells')
|
||||
}
|
||||
return cells
|
||||
}
|
||||
|
||||
function defaultCommand(args, json) {
|
||||
const result = spawnSync('gcloud', args, {
|
||||
encoding: 'utf8',
|
||||
stdio: ['ignore', 'pipe', 'pipe']
|
||||
})
|
||||
if (result.status !== 0) {
|
||||
throw new Error(`gcloud ${args.slice(0, 5).join(' ')} failed: ${result.stderr.trim()}`)
|
||||
}
|
||||
return json ? JSON.parse(result.stdout) : result.stdout.trim()
|
||||
}
|
||||
|
||||
function suppliedAdminToken(environment = process.env) {
|
||||
const token = environment.ORCA_RELAY_ADMIN_ID_TOKEN
|
||||
if (!token || token.length > 8_192 || !/^[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+$/.test(token)) {
|
||||
throw new Error('workflow did not supply a valid masked staging admin token')
|
||||
}
|
||||
return token
|
||||
}
|
||||
|
||||
async function responseJson(response, label) {
|
||||
const body = await response.json().catch(() => ({ error: `http_${response.status}` }))
|
||||
if (!response.ok) throw new Error(`${label} failed: ${body.error ?? response.status}`)
|
||||
return body
|
||||
}
|
||||
|
||||
function createAdminPost(deps) {
|
||||
const token = deps.adminToken()
|
||||
return async (origin, path, body) =>
|
||||
await responseJson(
|
||||
await deps.fetch(`${origin}${path}`, {
|
||||
method: 'POST',
|
||||
headers: { authorization: `Bearer ${token}`, 'content-type': 'application/json' },
|
||||
body: JSON.stringify(body),
|
||||
signal: AbortSignal.timeout(30_000)
|
||||
}),
|
||||
path
|
||||
)
|
||||
}
|
||||
|
||||
async function waitUntil(deps, label, operation, timeoutMs = WAKE_TIMEOUT_MS) {
|
||||
const deadline = deps.now() + timeoutMs
|
||||
let lastError
|
||||
while (deps.now() < deadline) {
|
||||
try {
|
||||
const result = await operation()
|
||||
if (result) return result
|
||||
} catch (error) {
|
||||
lastError = error
|
||||
}
|
||||
await deps.wait(POLL_INTERVAL_MS)
|
||||
}
|
||||
const detail = lastError instanceof Error ? `: ${lastError.message}` : ''
|
||||
throw new Error(`timed out waiting for ${label}${detail}`)
|
||||
}
|
||||
|
||||
async function checkHealth(deps, origin, path) {
|
||||
const response = await deps.fetch(`${origin}${path}`, { signal: AbortSignal.timeout(15_000) })
|
||||
const body = await response.json().catch(() => ({}))
|
||||
return response.ok && body.ok === true
|
||||
}
|
||||
|
||||
function sqlActivationPolicy(instance) {
|
||||
return String(instance.settings?.activationPolicy ?? '')
|
||||
}
|
||||
|
||||
function describeSql(deps) {
|
||||
return deps.commandJson([
|
||||
'sql',
|
||||
'instances',
|
||||
'describe',
|
||||
SQL_INSTANCE,
|
||||
'--project',
|
||||
PROJECT,
|
||||
'--format=json'
|
||||
])
|
||||
}
|
||||
|
||||
async function ensureSqlPolicy(deps, policy) {
|
||||
if (sqlActivationPolicy(describeSql(deps)) === policy) return false
|
||||
deps.command([
|
||||
'sql',
|
||||
'instances',
|
||||
'patch',
|
||||
SQL_INSTANCE,
|
||||
'--project',
|
||||
PROJECT,
|
||||
`--activation-policy=${policy}`,
|
||||
'--quiet'
|
||||
])
|
||||
await waitUntil(deps, `Cloud SQL activation policy ${policy}`, () =>
|
||||
sqlActivationPolicy(describeSql(deps)) === policy
|
||||
)
|
||||
return true
|
||||
}
|
||||
|
||||
function describeMig(deps, cell) {
|
||||
return deps.commandJson([
|
||||
'compute',
|
||||
'instance-groups',
|
||||
'managed',
|
||||
'describe',
|
||||
cell.migName,
|
||||
'--project',
|
||||
PROJECT,
|
||||
'--zone',
|
||||
cell.zone,
|
||||
'--format=json'
|
||||
])
|
||||
}
|
||||
|
||||
async function setMigSize(deps, cell, size) {
|
||||
const before = describeMig(deps, cell)
|
||||
if (Number(before.targetSize) !== size) {
|
||||
deps.command([
|
||||
'compute',
|
||||
'instance-groups',
|
||||
'managed',
|
||||
'resize',
|
||||
cell.migName,
|
||||
'--project',
|
||||
PROJECT,
|
||||
'--zone',
|
||||
cell.zone,
|
||||
`--size=${size}`,
|
||||
'--quiet'
|
||||
])
|
||||
}
|
||||
await waitUntil(deps, `${cell.cellId} size ${size}`, () => {
|
||||
const current = describeMig(deps, cell)
|
||||
return Number(current.targetSize) === size && current.status?.isStable === true
|
||||
})
|
||||
}
|
||||
|
||||
function activeRevisionName(service) {
|
||||
const active = (service.status?.traffic ?? []).filter((entry) => Number(entry.percent ?? 0) > 0)
|
||||
if (active.length !== 1 || Number(active[0].percent) !== 100 || !active[0].revisionName) {
|
||||
throw new Error('Cloud Run service must have exactly one active revision')
|
||||
}
|
||||
return active[0].revisionName
|
||||
}
|
||||
|
||||
function describeRunService(deps, name) {
|
||||
return deps.commandJson([
|
||||
'run',
|
||||
'services',
|
||||
'describe',
|
||||
name,
|
||||
'--project',
|
||||
PROJECT,
|
||||
'--region',
|
||||
REGION,
|
||||
'--format=json'
|
||||
])
|
||||
}
|
||||
|
||||
function describeRunRevision(deps, name) {
|
||||
return deps.commandJson([
|
||||
'run',
|
||||
'revisions',
|
||||
'describe',
|
||||
name,
|
||||
'--project',
|
||||
PROJECT,
|
||||
'--region',
|
||||
REGION,
|
||||
'--format=json'
|
||||
])
|
||||
}
|
||||
|
||||
function revisionMinimum(revision) {
|
||||
return Number(revision.metadata?.annotations?.['autoscaling.knative.dev/minScale'] ?? 0)
|
||||
}
|
||||
|
||||
async function ensureCloudRunScaleToZero(deps, service) {
|
||||
const before = describeRunService(deps, service.name)
|
||||
const activeRevision = describeRunRevision(deps, activeRevisionName(before))
|
||||
if (revisionMinimum(activeRevision) === 0) return false
|
||||
|
||||
const latestName = before.status?.latestReadyRevisionName
|
||||
const latest = latestName ? describeRunRevision(deps, latestName) : null
|
||||
if (!latest || revisionMinimum(latest) !== 0) {
|
||||
deps.command([
|
||||
'run',
|
||||
'services',
|
||||
'update',
|
||||
service.name,
|
||||
'--project',
|
||||
PROJECT,
|
||||
'--region',
|
||||
REGION,
|
||||
'--min-instances=0',
|
||||
'--quiet'
|
||||
])
|
||||
}
|
||||
deps.command([
|
||||
'run',
|
||||
'services',
|
||||
'update-traffic',
|
||||
service.name,
|
||||
'--project',
|
||||
PROJECT,
|
||||
'--region',
|
||||
REGION,
|
||||
'--to-latest',
|
||||
'--quiet'
|
||||
])
|
||||
await waitUntil(deps, `${service.name} scale-to-zero revision`, async () => {
|
||||
const current = describeRunService(deps, service.name)
|
||||
const revision = describeRunRevision(deps, activeRevisionName(current))
|
||||
return revisionMinimum(revision) === 0 && (await checkHealth(deps, service.healthOrigin, '/health'))
|
||||
})
|
||||
return true
|
||||
}
|
||||
|
||||
async function cellStatus(adminPost, cell) {
|
||||
const response = await adminPost(DIRECTOR_ORIGIN, '/v1/admin/cell-status', {
|
||||
v: 1,
|
||||
cellId: cell.cellId
|
||||
})
|
||||
if (!response.status || response.status.cellId !== cell.cellId) {
|
||||
throw new Error(`${cell.cellId} returned an invalid status`)
|
||||
}
|
||||
return response.status
|
||||
}
|
||||
|
||||
function assertQuiescent(status) {
|
||||
const active = {
|
||||
activityLeases: status.activityLeases,
|
||||
activityRequestUnits: status.activityRequestUnits,
|
||||
outgoingMigrations: status.outgoingMigrations,
|
||||
incomingMigrations: status.incomingMigrations,
|
||||
observedRequests: status.runtime?.observedRequests ?? 0
|
||||
}
|
||||
if (Object.values(active).some((value) => Number(value) !== 0)) {
|
||||
throw new Error(`${status.cellId} still has active Relay work: ${JSON.stringify(active)}`)
|
||||
}
|
||||
}
|
||||
|
||||
// `undefined` when the director predates the step-5 reserve census (its router 404s the route).
|
||||
async function reserveCells(deps, cells) {
|
||||
const token = deps.adminToken()
|
||||
const reserve = []
|
||||
for (const cell of cells) {
|
||||
const response = await deps.fetch(`${DIRECTOR_ORIGIN}/v1/admin/cell-admit-mode`, {
|
||||
method: 'POST',
|
||||
headers: { authorization: `Bearer ${token}`, 'content-type': 'application/json' },
|
||||
body: JSON.stringify({ v: 1, cellId: cell.cellId }),
|
||||
signal: AbortSignal.timeout(30_000)
|
||||
})
|
||||
const result = await response.json().catch(() => null)
|
||||
if (response.status === 404 && result === null) return undefined
|
||||
if (!response.ok || result?.v !== 1 || !['db', 'reserve'].includes(result.admitMode)) {
|
||||
throw new Error(`${cell.cellId} admit mode read failed: ${result?.error ?? response.status}`)
|
||||
}
|
||||
if (result.admitMode === 'reserve') reserve.push(cell.cellId)
|
||||
}
|
||||
return reserve
|
||||
}
|
||||
|
||||
// Under the monotonic selector, admission belongs to the selector, and the selector-era wake
|
||||
// restores it; so this sleep proves the fleet empty and never writes a cell's admission.
|
||||
async function sleepUnderSelector(deps, cells, adminPost, selector) {
|
||||
const reserve = await reserveCells(deps, cells)
|
||||
if (reserve?.length) {
|
||||
throw new Error(`staging sleep refuses reserve-mode cells: ${reserve.join(', ')}`)
|
||||
}
|
||||
for (const service of CLOUD_RUN_SERVICES) await ensureCloudRunScaleToZero(deps, service)
|
||||
const final = await Promise.all(cells.map((cell) => cellStatus(adminPost, cell)))
|
||||
for (const status of final) assertQuiescent(status)
|
||||
const after = await inspectAdmissionSelector(
|
||||
async (path, body) => await adminPost(DIRECTOR_ORIGIN, path, body)
|
||||
)
|
||||
if (after.selector.generation !== selector.generation) {
|
||||
throw new Error('admission selector changed during staging sleep')
|
||||
}
|
||||
// The flag workflow runs outside the staging lock, so a reserve flip can land mid-sleep.
|
||||
const lateReserve = await reserveCells(deps, cells)
|
||||
if (lateReserve?.length) {
|
||||
throw new Error(`staging sleep refuses reserve-mode cells: ${lateReserve.join(', ')}`)
|
||||
}
|
||||
await Promise.all(cells.map((cell) => setMigSize(deps, cell, 0)))
|
||||
await ensureSqlPolicy(deps, 'NEVER')
|
||||
deps.emit({
|
||||
event: 'staging_relay_slept',
|
||||
stoppedCells: cells.map((cell) => cell.cellId),
|
||||
selectorGeneration: selector.generation
|
||||
})
|
||||
}
|
||||
|
||||
async function setCellState(adminPost, cell, enabled) {
|
||||
await adminPost(DIRECTOR_ORIGIN, '/v1/admin/cell-state', {
|
||||
v: 1,
|
||||
cellId: cell.cellId,
|
||||
enabled
|
||||
})
|
||||
}
|
||||
|
||||
async function stagingStatus(deps, cells) {
|
||||
return {
|
||||
event: 'staging_relay_power_status',
|
||||
project: PROJECT,
|
||||
sqlActivationPolicy: sqlActivationPolicy(describeSql(deps)),
|
||||
cells: cells.map((cell) => ({
|
||||
cellId: cell.cellId,
|
||||
initiallyEnabled: cell.initiallyEnabled,
|
||||
targetSize: Number(describeMig(deps, cell).targetSize)
|
||||
})),
|
||||
cloudRun: CLOUD_RUN_SERVICES.map((service) => {
|
||||
const described = describeRunService(deps, service.name)
|
||||
const revision = describeRunRevision(deps, activeRevisionName(described))
|
||||
return { service: service.name, activeRevisionMinimum: revisionMinimum(revision) }
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
async function sleepStaging(deps, cells) {
|
||||
const sqlPolicy = sqlActivationPolicy(describeSql(deps))
|
||||
if (sqlPolicy === 'NEVER') {
|
||||
const runningCells = cells.filter((cell) => Number(describeMig(deps, cell).targetSize) !== 0)
|
||||
if (runningCells.length > 0) {
|
||||
// SQL-off plus running workers is an unknown partial state; never kill those workers blindly.
|
||||
throw new Error(
|
||||
`staging is partially asleep with running cells: ${runningCells.map((cell) => cell.cellId).join(', ')}`
|
||||
)
|
||||
}
|
||||
deps.emit({ event: 'staging_relay_sleep_reconciled', alreadyAsleep: true })
|
||||
return
|
||||
}
|
||||
|
||||
const adminPost = createAdminPost(deps)
|
||||
const initial = await Promise.all(cells.map((cell) => cellStatus(adminPost, cell)))
|
||||
for (const status of initial) assertQuiescent(status)
|
||||
const selector = await inspectAdmissionSelector(
|
||||
async (path, body) => await adminPost(DIRECTOR_ORIGIN, path, body)
|
||||
)
|
||||
if (selector.selector.generation > 0) {
|
||||
await sleepUnderSelector(deps, cells, adminPost, selector.selector)
|
||||
return
|
||||
}
|
||||
const previouslyEnabled = new Set(initial.filter((status) => status.enabled).map((status) => status.cellId))
|
||||
|
||||
for (const cell of cells) await setCellState(adminPost, cell, false)
|
||||
await deps.wait(15_000)
|
||||
try {
|
||||
const disabled = await Promise.all(cells.map((cell) => cellStatus(adminPost, cell)))
|
||||
for (const status of disabled) {
|
||||
if (status.enabled) throw new Error(`${status.cellId} admission did not disable`)
|
||||
assertQuiescent(status)
|
||||
}
|
||||
for (const service of CLOUD_RUN_SERVICES) await ensureCloudRunScaleToZero(deps, service)
|
||||
const final = await Promise.all(cells.map((cell) => cellStatus(adminPost, cell)))
|
||||
for (const status of final) assertQuiescent(status)
|
||||
} catch (error) {
|
||||
for (const cell of cells.filter((candidate) => previouslyEnabled.has(candidate.cellId))) {
|
||||
await setCellState(adminPost, cell, true).catch(() => undefined)
|
||||
}
|
||||
throw error
|
||||
}
|
||||
|
||||
await Promise.all(cells.map((cell) => setMigSize(deps, cell, 0)))
|
||||
await ensureSqlPolicy(deps, 'NEVER')
|
||||
deps.emit({ event: 'staging_relay_slept', stoppedCells: cells.map((cell) => cell.cellId) })
|
||||
}
|
||||
|
||||
async function wakeStaging(deps, cells, wakeCells) {
|
||||
await ensureSqlPolicy(deps, 'ALWAYS')
|
||||
await waitUntil(deps, 'staging director health', () => checkHealth(deps, DIRECTOR_ORIGIN, '/health'))
|
||||
for (const service of CLOUD_RUN_SERVICES) await ensureCloudRunScaleToZero(deps, service)
|
||||
|
||||
const adminPost = createAdminPost(deps)
|
||||
const selector = await inspectAdmissionSelector(
|
||||
async (path, body) => await adminPost(DIRECTOR_ORIGIN, path, body)
|
||||
)
|
||||
const selectorActive = selector.selector.generation > 0
|
||||
// Existing-only cells may still own live or dormant assignments, so a
|
||||
// selector-era wake restores the complete retained topology.
|
||||
const selected = selectorActive
|
||||
? cells
|
||||
: cells.filter((cell) => wakeCells === 'all' || cell.initiallyEnabled)
|
||||
await Promise.all(
|
||||
cells.map((cell) => setMigSize(deps, cell, selected.includes(cell) ? 1 : 0))
|
||||
)
|
||||
for (const cell of selected) {
|
||||
await waitUntil(deps, `${cell.cellId} health`, () => checkHealth(deps, cell.origin, '/health'))
|
||||
await waitUntil(deps, `${cell.cellId} readiness`, () => checkHealth(deps, cell.origin, '/ready'))
|
||||
}
|
||||
|
||||
for (const cell of cells) {
|
||||
if (selectorActive) {
|
||||
const status = await waitUntil(deps, `${cell.cellId} authenticated heartbeat`, async () => {
|
||||
const current = await cellStatus(adminPost, cell)
|
||||
return current.runtime?.heartbeatFresh && current.runtime.ready ? current : null
|
||||
})
|
||||
if (status.admissionState !== selectorCellState(selector.selector, cell.cellId)) {
|
||||
throw new Error(`${cell.cellId} admission does not match selector`)
|
||||
}
|
||||
continue
|
||||
}
|
||||
if (!selected.includes(cell)) {
|
||||
await setCellState(adminPost, cell, false)
|
||||
continue
|
||||
}
|
||||
const status = await waitUntil(deps, `${cell.cellId} authenticated heartbeat`, async () => {
|
||||
const current = await cellStatus(adminPost, cell)
|
||||
return current.runtime?.heartbeatFresh && current.runtime.ready ? current : null
|
||||
})
|
||||
if (cell.initiallyEnabled && !status.enabled) await setCellState(adminPost, cell, true)
|
||||
if (!cell.initiallyEnabled && status.enabled) await setCellState(adminPost, cell, false)
|
||||
}
|
||||
deps.emit({
|
||||
event: 'staging_relay_woke',
|
||||
runningCells: selected.map((cell) => cell.cellId),
|
||||
admissionCells: selectorActive
|
||||
? selector.selector.membership.general
|
||||
: selected.filter((cell) => cell.initiallyEnabled).map((cell) => cell.cellId)
|
||||
})
|
||||
}
|
||||
|
||||
export async function runStagingRelayPower(config, overrides = {}) {
|
||||
const deps = {
|
||||
command: overrides.command ?? ((args) => defaultCommand(args, false)),
|
||||
commandJson: overrides.commandJson ?? ((args) => defaultCommand(args, true)),
|
||||
fetch: overrides.fetch ?? fetch,
|
||||
adminToken: overrides.adminToken ?? suppliedAdminToken,
|
||||
emit: overrides.emit ?? ((event) => process.stdout.write(`${JSON.stringify(event)}\n`)),
|
||||
now: overrides.now ?? Date.now,
|
||||
wait: overrides.wait ?? ((ms) => new Promise((resolve) => setTimeout(resolve, ms)))
|
||||
}
|
||||
const cells = readStagingTopology(config.topologyFile)
|
||||
if (config.mode === 'status') deps.emit(await stagingStatus(deps, cells))
|
||||
else if (config.mode === 'sleep') await sleepStaging(deps, cells)
|
||||
else await wakeStaging(deps, cells, config.wakeCells)
|
||||
}
|
||||
|
||||
export async function main(argv = process.argv.slice(2)) {
|
||||
await runStagingRelayPower(parseArguments(argv))
|
||||
}
|
||||
|
||||
if (import.meta.url === pathToFileURL(process.argv[1]).href) {
|
||||
main().catch((error) => {
|
||||
process.stderr.write(`${error instanceof Error ? error.message : String(error)}\n`)
|
||||
process.exitCode = 1
|
||||
})
|
||||
}
|
||||
@@ -1,427 +0,0 @@
|
||||
import assert from 'node:assert/strict'
|
||||
import { mkdtempSync, writeFileSync } from 'node:fs'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { join } from 'node:path'
|
||||
import { test } from 'node:test'
|
||||
import {
|
||||
parseArguments,
|
||||
readStagingTopology,
|
||||
runStagingRelayPower
|
||||
} from './power-staging-relay.mjs'
|
||||
|
||||
function topologyFile(overrides = {}) {
|
||||
const directory = mkdtempSync(join(tmpdir(), 'staging-relay-power-'))
|
||||
const topology = {
|
||||
'staging-gce-c1': {
|
||||
mig_name: 'orca-cloud-staging-relay-gce-c1',
|
||||
zone: 'us-central1-b',
|
||||
origin: 'https://c1.relay-staging.onorca.dev',
|
||||
initially_enabled: true
|
||||
},
|
||||
'staging-gce-c2': {
|
||||
mig_name: 'orca-cloud-staging-relay-gce-c2',
|
||||
zone: 'us-central1-c',
|
||||
origin: 'https://c2.relay-staging.onorca.dev',
|
||||
initially_enabled: true
|
||||
},
|
||||
'staging-gce-c3': {
|
||||
mig_name: 'orca-cloud-staging-relay-gce-c3',
|
||||
zone: 'us-central1-a',
|
||||
origin: 'https://c3.relay-staging.onorca.dev',
|
||||
initially_enabled: false
|
||||
},
|
||||
'staging-gce-c4': {
|
||||
mig_name: 'orca-cloud-staging-relay-gce-c4',
|
||||
zone: 'asia-east2-a',
|
||||
origin: 'https://c4.relay-staging.onorca.dev',
|
||||
initially_enabled: false
|
||||
},
|
||||
...overrides
|
||||
}
|
||||
const file = join(directory, 'topology.json')
|
||||
writeFileSync(file, JSON.stringify(topology))
|
||||
return file
|
||||
}
|
||||
|
||||
function argumentConfig(file, mode, wakeCells = 'configured') {
|
||||
return parseArguments([
|
||||
'--mode',
|
||||
mode,
|
||||
'--wake-cells',
|
||||
wakeCells,
|
||||
'--topology-file',
|
||||
file
|
||||
])
|
||||
}
|
||||
|
||||
function response(body, status = 200) {
|
||||
return new Response(JSON.stringify(body), {
|
||||
status,
|
||||
headers: { 'content-type': 'application/json' }
|
||||
})
|
||||
}
|
||||
|
||||
function harness({
|
||||
sqlPolicy = 'ALWAYS',
|
||||
migSize = 1,
|
||||
observedRequests = 0,
|
||||
selectorGeneration = 0,
|
||||
admitModes
|
||||
} = {}) {
|
||||
const cells = new Map(
|
||||
['staging-gce-c1', 'staging-gce-c2', 'staging-gce-c3', 'staging-gce-c4'].map((cellId, index) => [
|
||||
cellId,
|
||||
{
|
||||
enabled: index < 2,
|
||||
targetSize: migSize,
|
||||
observedRequests,
|
||||
initiallyEnabled: index < 2
|
||||
}
|
||||
])
|
||||
)
|
||||
const revisions = new Map([
|
||||
['orca-cloud-relay-staging', { active: 'relay-00001', latest: 'relay-00001', min: 1 }],
|
||||
['orca-cloud-auth-staging', { active: 'auth-00001', latest: 'auth-00001', min: 1 }]
|
||||
])
|
||||
const revisionMinimums = new Map([
|
||||
['relay-00001', 1],
|
||||
['auth-00001', 1]
|
||||
])
|
||||
const commands = []
|
||||
const events = []
|
||||
const posts = []
|
||||
let activationPolicy = sqlPolicy
|
||||
let clock = 0
|
||||
|
||||
function cellForMig(name) {
|
||||
return [...cells.entries()].find(([, value], index) => {
|
||||
const suffix = `c${index + 1}`
|
||||
return name.endsWith(suffix) && value
|
||||
})
|
||||
}
|
||||
|
||||
function commandJson(args) {
|
||||
if (args[0] === 'sql') return { settings: { activationPolicy } }
|
||||
if (args[0] === 'compute') {
|
||||
const entry = cellForMig(args[4])
|
||||
return { targetSize: entry[1].targetSize, status: { isStable: true } }
|
||||
}
|
||||
if (args[0] === 'run' && args[1] === 'services') {
|
||||
const state = revisions.get(args[3])
|
||||
return {
|
||||
status: {
|
||||
latestReadyRevisionName: state.latest,
|
||||
traffic: [{ percent: 100, revisionName: state.active }]
|
||||
}
|
||||
}
|
||||
}
|
||||
if (args[0] === 'run' && args[1] === 'revisions') {
|
||||
return {
|
||||
metadata: {
|
||||
annotations: {
|
||||
'autoscaling.knative.dev/minScale': String(revisionMinimums.get(args[3]) ?? 0)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
throw new Error(`unexpected JSON command ${args.join(' ')}`)
|
||||
}
|
||||
|
||||
function command(args) {
|
||||
commands.push(args)
|
||||
if (args[0] === 'sql') {
|
||||
activationPolicy = args.find((arg) => arg.startsWith('--activation-policy='))?.split('=')[1]
|
||||
return
|
||||
}
|
||||
if (args[0] === 'compute') {
|
||||
const entry = cellForMig(args[4])
|
||||
entry[1].targetSize = Number(args.find((arg) => arg.startsWith('--size='))?.split('=')[1])
|
||||
return
|
||||
}
|
||||
if (args[0] === 'run' && args[1] === 'services' && args[2] === 'update') {
|
||||
const state = revisions.get(args[3])
|
||||
state.latest = `${args[3]}-power`
|
||||
revisionMinimums.set(state.latest, 0)
|
||||
return
|
||||
}
|
||||
if (args[0] === 'run' && args[1] === 'services' && args[2] === 'update-traffic') {
|
||||
const state = revisions.get(args[3])
|
||||
state.active = state.latest
|
||||
return
|
||||
}
|
||||
throw new Error(`unexpected command ${args.join(' ')}`)
|
||||
}
|
||||
|
||||
async function fetchImpl(url, options = {}) {
|
||||
const parsed = new URL(url)
|
||||
if (!options.method) return response({ ok: true })
|
||||
const body = JSON.parse(options.body)
|
||||
posts.push(parsed.pathname)
|
||||
if (parsed.pathname === '/v1/admin/cell-admit-mode') {
|
||||
// A pre-census director's router answers an unknown route with a plain-text 404.
|
||||
if (!admitModes) return new Response('404 Not Found', { status: 404 })
|
||||
return response({ v: 1, cellId: body.cellId, admitMode: admitModes[body.cellId] ?? 'db' })
|
||||
}
|
||||
if (parsed.pathname === '/v1/admin/cell-status') {
|
||||
const state = cells.get(body.cellId)
|
||||
const index = [...cells.keys()].indexOf(body.cellId)
|
||||
return response({
|
||||
v: 1,
|
||||
status: {
|
||||
cellId: body.cellId,
|
||||
enabled: state.enabled,
|
||||
admissionState:
|
||||
selectorGeneration > 0
|
||||
? index === 0
|
||||
? 'existing-only'
|
||||
: index === 1
|
||||
? 'migration-only'
|
||||
: index === 2
|
||||
? 'general'
|
||||
: 'migration-only'
|
||||
: state.enabled
|
||||
? 'general'
|
||||
: 'existing-only',
|
||||
assignments: 0,
|
||||
reservedRequests: 0,
|
||||
activityLeases: 0,
|
||||
activityRequestUnits: 0,
|
||||
outgoingMigrations: 0,
|
||||
incomingMigrations: 0,
|
||||
runtime: {
|
||||
ready: state.targetSize === 1,
|
||||
heartbeatFresh: state.targetSize === 1,
|
||||
observedRequests: state.observedRequests
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
if (parsed.pathname === '/v1/admin/admission-selector/status') {
|
||||
return response({
|
||||
v: 1,
|
||||
selector: {
|
||||
generation: selectorGeneration,
|
||||
attemptId: null,
|
||||
membership: {
|
||||
existingOnly:
|
||||
selectorGeneration > 0
|
||||
? ['staging-gce-c1']
|
||||
: [...cells]
|
||||
.filter(([, state]) => !state.enabled)
|
||||
.map(([cellId]) => cellId),
|
||||
migrationOnly:
|
||||
selectorGeneration > 0 ? ['staging-gce-c2', 'staging-gce-c4'] : [],
|
||||
general:
|
||||
selectorGeneration > 0
|
||||
? ['staging-gce-c3']
|
||||
: [...cells]
|
||||
.filter(([, state]) => state.enabled)
|
||||
.map(([cellId]) => cellId)
|
||||
}
|
||||
},
|
||||
intent: null
|
||||
})
|
||||
}
|
||||
if (parsed.pathname === '/v1/admin/cell-state') {
|
||||
cells.get(body.cellId).enabled = body.enabled
|
||||
return response({ ok: true })
|
||||
}
|
||||
throw new Error(`unexpected fetch ${parsed.pathname}`)
|
||||
}
|
||||
|
||||
return {
|
||||
cells,
|
||||
commands,
|
||||
events,
|
||||
posts,
|
||||
admitModes,
|
||||
deps: {
|
||||
command,
|
||||
commandJson,
|
||||
fetch: fetchImpl,
|
||||
adminToken: () => 'header.payload.signature',
|
||||
emit: (event) => events.push(event),
|
||||
now: () => clock,
|
||||
wait: async (ms) => {
|
||||
clock += ms
|
||||
}
|
||||
},
|
||||
sqlPolicy: () => activationPolicy
|
||||
}
|
||||
}
|
||||
|
||||
test('accepts only explicit staging power arguments and topology', () => {
|
||||
const file = topologyFile()
|
||||
assert.equal(argumentConfig(file, 'status').mode, 'status')
|
||||
assert.equal(readStagingTopology(file).at(-1).zone, 'asia-east2-a')
|
||||
assert.throws(() => argumentConfig(file, 'destroy'))
|
||||
assert.throws(() => parseArguments(['--mode', 'sleep', '--wake-cells', 'configured']))
|
||||
|
||||
const unsafe = topologyFile({
|
||||
'staging-gce-c1': {
|
||||
mig_name: 'orca-cloud-relay-gce-c1',
|
||||
zone: 'us-central1-a',
|
||||
origin: 'https://c1.relay.onorca.dev',
|
||||
initially_enabled: true
|
||||
}
|
||||
})
|
||||
assert.throws(() => readStagingTopology(unsafe), /unsafe/)
|
||||
|
||||
const unreviewedRegion = topologyFile({
|
||||
'staging-gce-c4': {
|
||||
mig_name: 'orca-cloud-staging-relay-gce-c4',
|
||||
zone: 'europe-west1-b',
|
||||
origin: 'https://c4.relay-staging.onorca.dev',
|
||||
initially_enabled: false
|
||||
}
|
||||
})
|
||||
assert.throws(() => readStagingTopology(unreviewedRegion), /unsafe zone/)
|
||||
})
|
||||
|
||||
test('refuses sleep before changing admission when a cell has active requests', async () => {
|
||||
const testHarness = harness({ observedRequests: 1 })
|
||||
await assert.rejects(
|
||||
runStagingRelayPower(argumentConfig(topologyFile(), 'sleep'), testHarness.deps),
|
||||
/still has active Relay work/
|
||||
)
|
||||
assert.equal(testHarness.commands.length, 0)
|
||||
assert.equal(testHarness.cells.get('staging-gce-c1').enabled, true)
|
||||
})
|
||||
|
||||
test('sleeps only after disabling admission and proving zero active work', async () => {
|
||||
const testHarness = harness()
|
||||
await runStagingRelayPower(argumentConfig(topologyFile(), 'sleep'), testHarness.deps)
|
||||
|
||||
assert.equal(testHarness.sqlPolicy(), 'NEVER')
|
||||
assert.deepEqual([...testHarness.cells.values()].map((cell) => cell.targetSize), [0, 0, 0, 0])
|
||||
assert.deepEqual([...testHarness.cells.values()].map((cell) => cell.enabled), [false, false, false, false])
|
||||
assert.equal(testHarness.events.at(-1).event, 'staging_relay_slept')
|
||||
assert.equal(
|
||||
testHarness.commands.filter((args) => args[0] === 'run' && args[2] === 'update').length,
|
||||
2
|
||||
)
|
||||
})
|
||||
|
||||
test('sleeps under the selector without writing any cell admission', async () => {
|
||||
for (const admitModes of [undefined, {}]) {
|
||||
const testHarness = harness({ selectorGeneration: 3, admitModes })
|
||||
const before = [...testHarness.cells.values()].map((cell) => cell.enabled)
|
||||
await runStagingRelayPower(argumentConfig(topologyFile(), 'sleep'), testHarness.deps)
|
||||
|
||||
assert.equal(testHarness.sqlPolicy(), 'NEVER')
|
||||
assert.deepEqual([...testHarness.cells.values()].map((cell) => cell.targetSize), [0, 0, 0, 0])
|
||||
assert.deepEqual([...testHarness.cells.values()].map((cell) => cell.enabled), before)
|
||||
assert.equal(testHarness.posts.includes('/v1/admin/cell-state'), false)
|
||||
assert.equal(testHarness.posts.includes('/v1/admin/cell-admit-mode'), true)
|
||||
assert.deepEqual(testHarness.events.at(-1), {
|
||||
event: 'staging_relay_slept',
|
||||
stoppedCells: ['staging-gce-c1', 'staging-gce-c2', 'staging-gce-c3', 'staging-gce-c4'],
|
||||
selectorGeneration: 3
|
||||
})
|
||||
}
|
||||
})
|
||||
|
||||
test('refuses a selector-era sleep while any cell is reserve or busy', async () => {
|
||||
const reserve = harness({ selectorGeneration: 3, admitModes: { 'staging-gce-c2': 'reserve' } })
|
||||
await assert.rejects(
|
||||
runStagingRelayPower(argumentConfig(topologyFile(), 'sleep'), reserve.deps),
|
||||
/refuses reserve-mode cells: staging-gce-c2/
|
||||
)
|
||||
assert.equal(reserve.commands.length, 0)
|
||||
|
||||
// A flip that lands after the first read, while Cloud Run scales down, still stops the resize.
|
||||
const late = harness({ selectorGeneration: 3, admitModes: {} })
|
||||
const run = late.deps.command
|
||||
late.deps.command = (args) => {
|
||||
if (args[0] === 'run' && args[2] === 'update-traffic') late.admitModes['staging-gce-c4'] = 'reserve'
|
||||
return run(args)
|
||||
}
|
||||
await assert.rejects(
|
||||
runStagingRelayPower(argumentConfig(topologyFile(), 'sleep'), late.deps),
|
||||
/refuses reserve-mode cells: staging-gce-c4/
|
||||
)
|
||||
assert.deepEqual([...late.cells.values()].map((cell) => cell.targetSize), [1, 1, 1, 1])
|
||||
assert.equal(late.sqlPolicy(), 'ALWAYS')
|
||||
|
||||
const busy = harness({ selectorGeneration: 3, observedRequests: 1 })
|
||||
await assert.rejects(
|
||||
runStagingRelayPower(argumentConfig(topologyFile(), 'sleep'), busy.deps),
|
||||
/still has active Relay work/
|
||||
)
|
||||
assert.equal(busy.commands.length, 0)
|
||||
assert.deepEqual([...busy.cells.values()].map((cell) => cell.targetSize), [1, 1, 1, 1])
|
||||
})
|
||||
|
||||
test('a selector-era sleep then wake restores the selector membership', async () => {
|
||||
const testHarness = harness({ selectorGeneration: 3 })
|
||||
const file = topologyFile()
|
||||
await runStagingRelayPower(argumentConfig(file, 'sleep'), testHarness.deps)
|
||||
for (const state of testHarness.cells.values()) state.observedRequests = 0
|
||||
await runStagingRelayPower(argumentConfig(file, 'wake'), testHarness.deps)
|
||||
|
||||
assert.equal(testHarness.sqlPolicy(), 'ALWAYS')
|
||||
assert.deepEqual([...testHarness.cells.values()].map((cell) => cell.targetSize), [1, 1, 1, 1])
|
||||
assert.equal(testHarness.posts.includes('/v1/admin/cell-state'), false)
|
||||
assert.deepEqual(testHarness.events.at(-1), {
|
||||
event: 'staging_relay_woke',
|
||||
runningCells: ['staging-gce-c1', 'staging-gce-c2', 'staging-gce-c3', 'staging-gce-c4'],
|
||||
admissionCells: ['staging-gce-c3']
|
||||
})
|
||||
})
|
||||
|
||||
test('refuses to terminate workers from an unknown partially asleep state', async () => {
|
||||
const testHarness = harness({ sqlPolicy: 'NEVER', migSize: 1 })
|
||||
await assert.rejects(
|
||||
runStagingRelayPower(argumentConfig(topologyFile(), 'sleep'), testHarness.deps),
|
||||
/partially asleep with running cells/
|
||||
)
|
||||
assert.equal(testHarness.commands.length, 0)
|
||||
assert.deepEqual([...testHarness.cells.values()].map((cell) => cell.targetSize), [1, 1, 1, 1])
|
||||
})
|
||||
|
||||
test('wakes SQL and configured cells while leaving the candidate off and disabled', async () => {
|
||||
const testHarness = harness({ sqlPolicy: 'NEVER', migSize: 0 })
|
||||
for (const cell of testHarness.cells.values()) cell.enabled = false
|
||||
for (const state of testHarness.cells.values()) state.observedRequests = 0
|
||||
await runStagingRelayPower(argumentConfig(topologyFile(), 'wake'), testHarness.deps)
|
||||
|
||||
assert.equal(testHarness.sqlPolicy(), 'ALWAYS')
|
||||
assert.deepEqual([...testHarness.cells.values()].map((cell) => cell.targetSize), [1, 1, 0, 0])
|
||||
assert.deepEqual([...testHarness.cells.values()].map((cell) => cell.enabled), [true, true, false, false])
|
||||
assert.deepEqual(testHarness.events.at(-1), {
|
||||
event: 'staging_relay_woke',
|
||||
runningCells: ['staging-gce-c1', 'staging-gce-c2'],
|
||||
admissionCells: ['staging-gce-c1', 'staging-gce-c2']
|
||||
})
|
||||
})
|
||||
|
||||
test('wakes every retained cell without rewriting selector-era admission', async () => {
|
||||
const testHarness = harness({
|
||||
sqlPolicy: 'NEVER',
|
||||
migSize: 0,
|
||||
selectorGeneration: 1
|
||||
})
|
||||
const states = [...testHarness.cells.values()]
|
||||
states[0].enabled = false
|
||||
states[1].enabled = true
|
||||
states[2].enabled = true
|
||||
states[3].enabled = true
|
||||
await runStagingRelayPower(argumentConfig(topologyFile(), 'wake'), testHarness.deps)
|
||||
|
||||
assert.deepEqual(states.map((cell) => cell.targetSize), [1, 1, 1, 1])
|
||||
assert.deepEqual(states.map((cell) => cell.enabled), [false, true, true, true])
|
||||
assert.deepEqual(testHarness.events.at(-1), {
|
||||
event: 'staging_relay_woke',
|
||||
runningCells: ['staging-gce-c1', 'staging-gce-c2', 'staging-gce-c3', 'staging-gce-c4'],
|
||||
admissionCells: ['staging-gce-c3']
|
||||
})
|
||||
})
|
||||
|
||||
test('status is read-only and reports the current billable floor controls', async () => {
|
||||
const testHarness = harness()
|
||||
await runStagingRelayPower(argumentConfig(topologyFile(), 'status'), testHarness.deps)
|
||||
assert.equal(testHarness.commands.length, 0)
|
||||
assert.equal(testHarness.events[0].project, 'onorca-cloud-staging')
|
||||
assert.equal(testHarness.events[0].sqlActivationPolicy, 'ALWAYS')
|
||||
assert.equal(testHarness.events[0].cells.length, 4)
|
||||
})
|
||||
@@ -1,86 +0,0 @@
|
||||
import { readFileSync, writeFileSync } from 'node:fs'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
|
||||
function argumentsFrom(argv) {
|
||||
const values = {}
|
||||
for (let index = 0; index < argv.length; index += 2) {
|
||||
const key = argv[index]
|
||||
const value = argv[index + 1]
|
||||
if (!key?.startsWith('--') || value === undefined) throw new Error('invalid arguments')
|
||||
values[key.slice(2)] = value
|
||||
}
|
||||
for (const key of ['current-json', 'topology-json', 'output', 'cell-ids', 'image-digest']) {
|
||||
if (!values[key]) throw new Error(`missing --${key}`)
|
||||
}
|
||||
return values
|
||||
}
|
||||
|
||||
// Production Asia pools sit 176 ms from Cloud SQL and run at 16; staging C4 and the production
|
||||
// US cells at the Asia shape keep the default 10.
|
||||
const DATABASE_POOL_MAX = {
|
||||
production: { 'asia-east2': 16, 'us-central1': 10 },
|
||||
staging: { 'asia-east2': 10 }
|
||||
}
|
||||
|
||||
export function prepareRelayAsiaDirectorCells({ currentCells, topology, cellIds, imageDigest }) {
|
||||
if (!Array.isArray(currentCells) || !topology || Array.isArray(topology)) {
|
||||
throw new Error('director inputs are invalid')
|
||||
}
|
||||
const additions = cellIds.split(',').map((value) => value.trim()).filter(Boolean)
|
||||
if (
|
||||
additions.length === 0 ||
|
||||
new Set(additions).size !== additions.length
|
||||
) throw new Error('director Asia additions are invalid')
|
||||
const currentIds = new Set(currentCells.map((cell) => cell.id))
|
||||
if (currentIds.size !== currentCells.length) throw new Error('current director cells contain duplicates')
|
||||
const normalizedCurrent = currentCells.map((cell) => ({
|
||||
...cell,
|
||||
region: cell.region ?? 'us-central1'
|
||||
}))
|
||||
const desiredCells = additions.map((cellId) => {
|
||||
const cell = topology[cellId]
|
||||
const pools = DATABASE_POOL_MAX[cellId.split('-')[0]] ?? {}
|
||||
if (
|
||||
!cell ||
|
||||
!Object.hasOwn(pools, cell.region) ||
|
||||
cell.capacity_requests !== 6_000 ||
|
||||
cell.database_pool_max !== pools[cell.region] ||
|
||||
cell.connection_hard_cap !== 3_000 ||
|
||||
cell.connection_unobserved_bound !== 60 ||
|
||||
cell.initially_enabled !== false ||
|
||||
cell.image?.split('@')[1] !== imageDigest
|
||||
) throw new Error(`${cellId} state output does not match the reviewed Asia shape`)
|
||||
return {
|
||||
id: cellId,
|
||||
url: cell.origin,
|
||||
capacityRequests: cell.capacity_requests,
|
||||
region: cell.region,
|
||||
initiallyEnabled: false,
|
||||
connectionHardCap: cell.connection_hard_cap,
|
||||
connectionUnobservedBound: cell.connection_unobserved_bound
|
||||
}
|
||||
})
|
||||
const desiredById = new Map(desiredCells.map((cell) => [cell.id, cell]))
|
||||
for (const current of normalizedCurrent) {
|
||||
const desired = desiredById.get(current.id)
|
||||
if (!desired) continue
|
||||
for (const [key, value] of Object.entries(desired)) {
|
||||
if (current[key] !== value) {
|
||||
throw new Error(`${current.id} director configuration differs from the reviewed Asia shape`)
|
||||
}
|
||||
}
|
||||
}
|
||||
const configured = new Set(normalizedCurrent.map((cell) => cell.id))
|
||||
return [...normalizedCurrent, ...desiredCells.filter((cell) => !configured.has(cell.id))]
|
||||
}
|
||||
|
||||
if (process.argv[1] === fileURLToPath(import.meta.url)) {
|
||||
const values = argumentsFrom(process.argv.slice(2))
|
||||
const result = prepareRelayAsiaDirectorCells({
|
||||
currentCells: JSON.parse(readFileSync(values['current-json'], 'utf8')),
|
||||
topology: JSON.parse(readFileSync(values['topology-json'], 'utf8')),
|
||||
cellIds: values['cell-ids'],
|
||||
imageDigest: values['image-digest']
|
||||
})
|
||||
writeFileSync(values.output, `${JSON.stringify(result)}\n`, { mode: 0o600 })
|
||||
}
|
||||
@@ -1,153 +0,0 @@
|
||||
import assert from 'node:assert/strict'
|
||||
import { test } from 'node:test'
|
||||
import { prepareRelayAsiaDirectorCells } from './prepare-relay-asia-director-cells.mjs'
|
||||
|
||||
const digest = `sha256:${'a'.repeat(64)}`
|
||||
const topologyCell = (ordinal, zone) => ({
|
||||
origin: `https://c${ordinal}.relay.onorca.dev`, region: 'asia-east2', zone,
|
||||
capacity_requests: 6_000, database_pool_max: 16,
|
||||
connection_hard_cap: 3_000, connection_unobserved_bound: 60,
|
||||
initially_enabled: false,
|
||||
image: `us-central1-docker.pkg.dev/onorca-cloud/orca-cloud/relay@${digest}`
|
||||
})
|
||||
|
||||
test('preserves current order, defaults predecessor regions, and appends exact Asia cells', () => {
|
||||
const current = [{
|
||||
id: 'production-gce-c1', url: 'https://c1.relay.onorca.dev',
|
||||
capacityRequests: 4_000, initiallyEnabled: false
|
||||
}]
|
||||
const result = prepareRelayAsiaDirectorCells({
|
||||
currentCells: current,
|
||||
topology: {
|
||||
'production-gce-c27': topologyCell(27, 'asia-east2-a'),
|
||||
'production-gce-c28': topologyCell(28, 'asia-east2-b'),
|
||||
'production-gce-c29': topologyCell(29, 'asia-east2-c')
|
||||
},
|
||||
cellIds: 'production-gce-c27,production-gce-c28,production-gce-c29',
|
||||
imageDigest: digest
|
||||
})
|
||||
assert.equal(result[0].region, 'us-central1')
|
||||
assert.deepEqual(result.slice(1).map(({ id }) => id), [
|
||||
'production-gce-c27', 'production-gce-c28', 'production-gce-c29'
|
||||
])
|
||||
assert.ok(result.slice(1).every((cell) =>
|
||||
cell.region === 'asia-east2' && cell.initiallyEnabled === false &&
|
||||
cell.connectionHardCap === 3_000
|
||||
))
|
||||
})
|
||||
|
||||
test('is idempotent for an exact existing Asia cell and rejects director drift', () => {
|
||||
const topology = { 'production-gce-c27': topologyCell(27, 'asia-east2-a') }
|
||||
const current = prepareRelayAsiaDirectorCells({
|
||||
currentCells: [], topology, cellIds: 'production-gce-c27', imageDigest: digest
|
||||
})
|
||||
assert.deepEqual(prepareRelayAsiaDirectorCells({
|
||||
currentCells: current, topology, cellIds: 'production-gce-c27', imageDigest: digest
|
||||
}), current)
|
||||
assert.throws(() => prepareRelayAsiaDirectorCells({
|
||||
currentCells: [{ ...current[0], capacityRequests: 5_999 }],
|
||||
topology, cellIds: 'production-gce-c27', imageDigest: digest
|
||||
}), /director configuration differs/)
|
||||
})
|
||||
|
||||
test('appends C30 after the configured launch cells without touching them', () => {
|
||||
const launch = prepareRelayAsiaDirectorCells({
|
||||
currentCells: [],
|
||||
topology: {
|
||||
'production-gce-c27': topologyCell(27, 'asia-east2-a'),
|
||||
'production-gce-c28': topologyCell(28, 'asia-east2-b'),
|
||||
'production-gce-c29': topologyCell(29, 'asia-east2-c')
|
||||
},
|
||||
cellIds: 'production-gce-c27,production-gce-c28,production-gce-c29',
|
||||
imageDigest: digest
|
||||
})
|
||||
const result = prepareRelayAsiaDirectorCells({
|
||||
currentCells: launch,
|
||||
topology: { 'production-gce-c30': topologyCell(30, 'asia-east2-a') },
|
||||
cellIds: 'production-gce-c30',
|
||||
imageDigest: digest
|
||||
})
|
||||
assert.deepEqual(result.slice(0, 3), launch)
|
||||
assert.deepEqual(result[3], {
|
||||
id: 'production-gce-c30', url: 'https://c30.relay.onorca.dev', capacityRequests: 6_000,
|
||||
region: 'asia-east2', initiallyEnabled: false, connectionHardCap: 3_000,
|
||||
connectionUnobservedBound: 60
|
||||
})
|
||||
})
|
||||
|
||||
test('checks C30 against its own digest, not the launch cells\' digest', () => {
|
||||
const c30Digest = `sha256:${'b'.repeat(64)}`
|
||||
const c30 = {
|
||||
...topologyCell(30, 'asia-east2-a'),
|
||||
image: `us-central1-docker.pkg.dev/onorca-cloud/orca-cloud/relay@${c30Digest}`
|
||||
}
|
||||
const launch = [27, 28, 29].map((ordinal) => ({
|
||||
id: `production-gce-c${ordinal}`, url: `https://c${ordinal}.relay.onorca.dev`,
|
||||
capacityRequests: 6_000, region: 'asia-east2', initiallyEnabled: false,
|
||||
connectionHardCap: 3_000, connectionUnobservedBound: 60
|
||||
}))
|
||||
assert.equal(prepareRelayAsiaDirectorCells({
|
||||
currentCells: launch, topology: { 'production-gce-c30': c30 },
|
||||
cellIds: 'production-gce-c30', imageDigest: c30Digest
|
||||
}).length, 4)
|
||||
assert.throws(() => prepareRelayAsiaDirectorCells({
|
||||
currentCells: launch, topology: { 'production-gce-c30': c30 },
|
||||
cellIds: 'production-gce-c30', imageDigest: digest
|
||||
}), /does not match/)
|
||||
})
|
||||
|
||||
test('pins the Asia pool per environment', () => {
|
||||
const staging = { ...topologyCell(4, 'asia-east2-a'), database_pool_max: 10 }
|
||||
assert.equal(prepareRelayAsiaDirectorCells({
|
||||
currentCells: [], topology: { 'staging-gce-c4': staging },
|
||||
cellIds: 'staging-gce-c4', imageDigest: digest
|
||||
}).length, 1)
|
||||
assert.throws(() => prepareRelayAsiaDirectorCells({
|
||||
currentCells: [], topology: { 'staging-gce-c4': { ...staging, database_pool_max: 16 } },
|
||||
cellIds: 'staging-gce-c4', imageDigest: digest
|
||||
}), /does not match/)
|
||||
assert.throws(() => prepareRelayAsiaDirectorCells({
|
||||
currentCells: [],
|
||||
topology: { 'production-gce-c30': { ...topologyCell(30, 'asia-east2-a'), database_pool_max: 10 } },
|
||||
cellIds: 'production-gce-c30', imageDigest: digest
|
||||
}), /does not match/)
|
||||
})
|
||||
|
||||
test('rejects a mismatching topology state output', () => {
|
||||
const wrong = topologyCell(27, 'asia-east2-a')
|
||||
wrong.database_pool_max = 20
|
||||
assert.throws(() => prepareRelayAsiaDirectorCells({
|
||||
currentCells: [], topology: { 'production-gce-c27': wrong },
|
||||
cellIds: 'production-gce-c27', imageDigest: digest
|
||||
}), /does not match/)
|
||||
})
|
||||
|
||||
test('appends a US cell at the Asia shape with the default pool, never the Asia one', () => {
|
||||
const usCell = {
|
||||
...topologyCell(32, 'us-central1-a'), region: 'us-central1', database_pool_max: 10
|
||||
}
|
||||
const current = [{
|
||||
id: 'production-gce-c26', url: 'https://c26.relay.onorca.dev', capacityRequests: 4_000,
|
||||
initiallyEnabled: false, connectionHardCap: 1_000, connectionUnobservedBound: 60
|
||||
}]
|
||||
const result = prepareRelayAsiaDirectorCells({
|
||||
currentCells: current, topology: { 'production-gce-c32': usCell },
|
||||
cellIds: 'production-gce-c32', imageDigest: digest
|
||||
})
|
||||
assert.deepEqual(result.at(-1), {
|
||||
id: 'production-gce-c32', url: 'https://c32.relay.onorca.dev', capacityRequests: 6_000,
|
||||
region: 'us-central1', initiallyEnabled: false, connectionHardCap: 3_000,
|
||||
connectionUnobservedBound: 60
|
||||
})
|
||||
for (const drift of [{ database_pool_max: 16 }, { region: 'europe-west1' }]) {
|
||||
assert.throws(() => prepareRelayAsiaDirectorCells({
|
||||
currentCells: current, topology: { 'production-gce-c32': { ...usCell, ...drift } },
|
||||
cellIds: 'production-gce-c32', imageDigest: digest
|
||||
}), /reviewed Asia shape/, JSON.stringify(drift))
|
||||
}
|
||||
// Staging has no US cell at this shape.
|
||||
assert.throws(() => prepareRelayAsiaDirectorCells({
|
||||
currentCells: [], topology: { 'staging-gce-c4': { ...usCell, database_pool_max: 10 } },
|
||||
cellIds: 'staging-gce-c4', imageDigest: digest
|
||||
}), /reviewed Asia shape/)
|
||||
})
|
||||
@@ -1,140 +0,0 @@
|
||||
import { readFileSync } from 'node:fs'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
|
||||
const BOOT_IMAGE = 'https://www.googleapis.com/compute/v1/projects/cos-cloud/global/images/cos-121-18867-638-13'
|
||||
const SHAPES = {
|
||||
staging: {
|
||||
project: 'onorca-cloud-staging',
|
||||
databasePoolMax: { 'asia-east2': 10 },
|
||||
cells: { 'staging-gce-c4': 'asia-east2-a' },
|
||||
waves: [['staging-gce-c4']]
|
||||
},
|
||||
production: {
|
||||
project: 'onorca-cloud',
|
||||
// 16 covers asia-east2's round trip to us-central1 Postgres; US cells keep the default.
|
||||
databasePoolMax: { 'asia-east2': 16, 'us-central1': 10 },
|
||||
cells: {
|
||||
'production-gce-c27': 'asia-east2-a',
|
||||
'production-gce-c28': 'asia-east2-b',
|
||||
'production-gce-c29': 'asia-east2-c',
|
||||
'production-gce-c30': 'asia-east2-a',
|
||||
'production-gce-c31': 'asia-east2-b',
|
||||
'production-gce-c32': 'us-central1-a',
|
||||
'production-gce-c33': 'us-central1-b',
|
||||
'production-gce-c34': 'asia-east2-c',
|
||||
'production-gce-c35': 'asia-east2-a'
|
||||
},
|
||||
// The launch set, then each later additive cell; a plan targets one wave, never live cells.
|
||||
waves: [
|
||||
['production-gce-c27', 'production-gce-c28', 'production-gce-c29'],
|
||||
['production-gce-c30'],
|
||||
['production-gce-c31'],
|
||||
// Declared together, so they plan together: a lone C32 plan would hit C33's missing template.
|
||||
['production-gce-c32', 'production-gce-c33'],
|
||||
['production-gce-c34'],
|
||||
['production-gce-c35']
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
function argumentsFrom(argv) {
|
||||
const values = {}
|
||||
for (let index = 0; index < argv.length; index += 2) {
|
||||
const key = argv[index]
|
||||
const value = argv[index + 1]
|
||||
if (!key?.startsWith('--') || value === undefined) throw new Error('invalid arguments')
|
||||
values[key.slice(2)] = value
|
||||
}
|
||||
for (const key of ['existing-json', 'environment', 'cell-ids', 'image']) {
|
||||
if (!values[key]) throw new Error(`missing --${key}`)
|
||||
}
|
||||
return values
|
||||
}
|
||||
|
||||
function canonical(value) {
|
||||
if (Array.isArray(value)) return value.map(canonical)
|
||||
if (value && typeof value === 'object') {
|
||||
return Object.fromEntries(Object.keys(value).sort().map((key) => [key, canonical(value[key])]))
|
||||
}
|
||||
return value
|
||||
}
|
||||
|
||||
export function prepareRelayAsiaTopologyInput({
|
||||
existingCells,
|
||||
existingAdditionalRegions,
|
||||
environment,
|
||||
cellIds,
|
||||
image
|
||||
}) {
|
||||
const shape = SHAPES[environment]
|
||||
if (!shape) throw new Error('invalid environment')
|
||||
const requested = cellIds.split(',').map((value) => value.trim()).filter(Boolean).sort()
|
||||
const expected = shape.waves
|
||||
.map((wave) => [...wave].sort())
|
||||
.find((wave) => JSON.stringify(wave) === JSON.stringify(requested))
|
||||
if (new Set(requested).size !== requested.length || !expected) {
|
||||
throw new Error('cell IDs do not match the reviewed Asia topology')
|
||||
}
|
||||
const prefix = `us-central1-docker.pkg.dev/${shape.project}/orca-cloud/relay@sha256:`
|
||||
if (!image.startsWith(prefix) || !/sha256:[a-f0-9]{64}$/.test(image)) {
|
||||
throw new Error('image is not the environment Relay image pinned by digest')
|
||||
}
|
||||
if (!existingCells || Array.isArray(existingCells) || typeof existingCells !== 'object') {
|
||||
throw new Error('existing Relay cells must be an object')
|
||||
}
|
||||
if (
|
||||
!existingAdditionalRegions ||
|
||||
Array.isArray(existingAdditionalRegions) ||
|
||||
typeof existingAdditionalRegions !== 'object' ||
|
||||
JSON.stringify(canonical(existingAdditionalRegions)) !==
|
||||
JSON.stringify(canonical({ 'asia-east2': '10.42.1.0/24' }))
|
||||
) {
|
||||
throw new Error('Asia subnet must be committed before topology planning')
|
||||
}
|
||||
const additions = Object.fromEntries(expected.map((cellId) => {
|
||||
const hostname = cellId.split('-').at(-1)
|
||||
const zone = shape.cells[cellId]
|
||||
const region = zone.slice(0, zone.lastIndexOf('-'))
|
||||
return [cellId, {
|
||||
hostname,
|
||||
region,
|
||||
zone,
|
||||
machine_type: 'e2-standard-4',
|
||||
boot_disk_gb: 30,
|
||||
boot_image: BOOT_IMAGE,
|
||||
capacity_requests: 6_000,
|
||||
database_pool_max: shape.databasePoolMax[region],
|
||||
image,
|
||||
initially_enabled: false,
|
||||
connection_hard_cap: 3_000,
|
||||
connection_unobserved_bound: 60
|
||||
}]
|
||||
}))
|
||||
for (const cellId of expected) {
|
||||
if (!existingCells[cellId]) {
|
||||
throw new Error('Asia cells must be committed before topology planning')
|
||||
}
|
||||
if (
|
||||
JSON.stringify(canonical(existingCells[cellId])) !==
|
||||
JSON.stringify(canonical(additions[cellId]))
|
||||
) {
|
||||
throw new Error('committed Asia cell differs from the reviewed topology')
|
||||
}
|
||||
}
|
||||
return {
|
||||
relay_gce_additional_region_subnetwork_cidrs: existingAdditionalRegions,
|
||||
relay_gce_cells: existingCells
|
||||
}
|
||||
}
|
||||
|
||||
if (process.argv[1] === fileURLToPath(import.meta.url)) {
|
||||
const values = argumentsFrom(process.argv.slice(2))
|
||||
const existing = JSON.parse(readFileSync(values['existing-json'], 'utf8'))
|
||||
prepareRelayAsiaTopologyInput({
|
||||
existingCells: existing.relay_gce_cells,
|
||||
existingAdditionalRegions: existing.relay_gce_additional_region_subnetwork_cidrs,
|
||||
environment: values.environment,
|
||||
cellIds: values['cell-ids'],
|
||||
image: values.image
|
||||
})
|
||||
}
|
||||
@@ -1,211 +0,0 @@
|
||||
import assert from 'node:assert/strict'
|
||||
import { readFileSync } from 'node:fs'
|
||||
import { test } from 'node:test'
|
||||
import { prepareRelayAsiaTopologyInput } from './prepare-relay-asia-topology-input.mjs'
|
||||
|
||||
const image = `us-central1-docker.pkg.dev/onorca-cloud/orca-cloud/relay@sha256:${'a'.repeat(64)}`
|
||||
|
||||
const additionalRegions = { 'asia-east2': '10.42.1.0/24' }
|
||||
|
||||
const productionCells = () => Object.fromEntries([
|
||||
[27, 'asia-east2-a'],
|
||||
[28, 'asia-east2-b'],
|
||||
[29, 'asia-east2-c'],
|
||||
[30, 'asia-east2-a'],
|
||||
[31, 'asia-east2-b'],
|
||||
[32, 'us-central1-a'],
|
||||
[33, 'us-central1-b'],
|
||||
[34, 'asia-east2-c'],
|
||||
[35, 'asia-east2-a']
|
||||
].map(([ordinal, zone]) => [`production-gce-c${ordinal}`, {
|
||||
hostname: `c${ordinal}`, region: zone.slice(0, -2), zone,
|
||||
machine_type: 'e2-standard-4', boot_disk_gb: 30,
|
||||
boot_image: 'https://www.googleapis.com/compute/v1/projects/cos-cloud/global/images/cos-121-18867-638-13',
|
||||
capacity_requests: 6_000, database_pool_max: zone.startsWith('asia-') ? 16 : 10, image,
|
||||
initially_enabled: false,
|
||||
connection_hard_cap: 3_000, connection_unobserved_bound: 60
|
||||
}]))
|
||||
|
||||
test('accepts the exact production topology only after it is durably committed', () => {
|
||||
const existing = {
|
||||
'production-gce-c26': { hostname: 'c26', image: 'existing' },
|
||||
...productionCells()
|
||||
}
|
||||
const result = prepareRelayAsiaTopologyInput({ existingCells: existing,
|
||||
existingAdditionalRegions: additionalRegions, environment: 'production',
|
||||
cellIds: 'production-gce-c27,production-gce-c28,production-gce-c29', image })
|
||||
assert.equal(result.relay_gce_cells, existing)
|
||||
assert.equal(result.relay_gce_additional_region_subnetwork_cidrs, additionalRegions)
|
||||
assert.deepEqual(existing['production-gce-c27'], {
|
||||
hostname: 'c27', region: 'asia-east2', zone: 'asia-east2-a',
|
||||
machine_type: 'e2-standard-4', boot_disk_gb: 30,
|
||||
boot_image: 'https://www.googleapis.com/compute/v1/projects/cos-cloud/global/images/cos-121-18867-638-13',
|
||||
capacity_requests: 6_000, database_pool_max: 16, image, initially_enabled: false,
|
||||
connection_hard_cap: 3_000, connection_unobserved_bound: 60
|
||||
})
|
||||
})
|
||||
|
||||
test('accepts the additive C30 wave without re-planning the launch cells', () => {
|
||||
const result = prepareRelayAsiaTopologyInput({ existingCells: productionCells(),
|
||||
existingAdditionalRegions: additionalRegions, environment: 'production',
|
||||
cellIds: 'production-gce-c30', image })
|
||||
assert.equal(result.relay_gce_cells['production-gce-c30'].zone, 'asia-east2-a')
|
||||
})
|
||||
|
||||
test('accepts the additive C31 wave in the next zone of the rotation', () => {
|
||||
const result = prepareRelayAsiaTopologyInput({ existingCells: productionCells(),
|
||||
existingAdditionalRegions: additionalRegions, environment: 'production',
|
||||
cellIds: 'production-gce-c31', image })
|
||||
assert.equal(result.relay_gce_cells['production-gce-c31'].zone, 'asia-east2-b')
|
||||
})
|
||||
|
||||
test('accepts the additive C34 spare wave in asia-east2-c at the Asia pool', () => {
|
||||
const result = prepareRelayAsiaTopologyInput({ existingCells: productionCells(),
|
||||
existingAdditionalRegions: additionalRegions, environment: 'production',
|
||||
cellIds: 'production-gce-c34', image })
|
||||
assert.equal(result.relay_gce_cells['production-gce-c34'].zone, 'asia-east2-c')
|
||||
assert.equal(result.relay_gce_cells['production-gce-c34'].database_pool_max, 16)
|
||||
})
|
||||
|
||||
test('accepts the additive C35 spare wave in asia-east2-a at the Asia pool', () => {
|
||||
const result = prepareRelayAsiaTopologyInput({ existingCells: productionCells(),
|
||||
existingAdditionalRegions: additionalRegions, environment: 'production',
|
||||
cellIds: 'production-gce-c35', image })
|
||||
assert.equal(result.relay_gce_cells['production-gce-c35'].zone, 'asia-east2-a')
|
||||
assert.equal(result.relay_gce_cells['production-gce-c35'].database_pool_max, 16)
|
||||
})
|
||||
|
||||
test('accepts the additive US C32+C33 wave at the default pool only', () => {
|
||||
const cellIds = 'production-gce-c32,production-gce-c33'
|
||||
for (const [cellId, zone] of [
|
||||
['production-gce-c32', 'us-central1-a'], ['production-gce-c33', 'us-central1-b']
|
||||
]) {
|
||||
const result = prepareRelayAsiaTopologyInput({ existingCells: productionCells(),
|
||||
existingAdditionalRegions: additionalRegions, environment: 'production', cellIds, image })
|
||||
assert.equal(result.relay_gce_cells[cellId].zone, zone)
|
||||
assert.equal(result.relay_gce_cells[cellId].region, 'us-central1')
|
||||
// Copying the Asia pool onto a US cell is drift, not the reviewed shape.
|
||||
const asiaPool = productionCells()
|
||||
asiaPool[cellId].database_pool_max = 16
|
||||
assert.throws(() => prepareRelayAsiaTopologyInput({ existingCells: asiaPool,
|
||||
existingAdditionalRegions: additionalRegions, environment: 'production', cellIds, image
|
||||
}), /differs from the reviewed topology/)
|
||||
const asiaRegion = productionCells()
|
||||
asiaRegion[cellId].region = 'asia-east2'
|
||||
assert.throws(() => prepareRelayAsiaTopologyInput({ existingCells: asiaRegion,
|
||||
existingAdditionalRegions: additionalRegions, environment: 'production', cellIds, image
|
||||
}), /differs from the reviewed topology/)
|
||||
}
|
||||
})
|
||||
|
||||
// Reads the committed file so a reviewed-shape constant cannot drift from what the plan reads.
|
||||
test('matches every committed production Asia cell entry', () => {
|
||||
const tfvars = readFileSync(
|
||||
new URL('../../infra/terraform/environments/production.tfvars', import.meta.url),
|
||||
'utf8'
|
||||
)
|
||||
const committed = {}
|
||||
for (const cellId of Object.keys(productionCells())) {
|
||||
const start = tfvars.indexOf(` "${cellId}" = {`)
|
||||
assert.notEqual(start, -1, `${cellId} is not committed`)
|
||||
const body = tfvars.slice(start, tfvars.indexOf('\n }', start))
|
||||
const cell = {}
|
||||
for (const [, key, raw] of body.matchAll(/^\s+([a-z_]+)\s+=\s+("[^"]*"|\S+)/gm)) {
|
||||
cell[key] = raw.startsWith('"') ? raw.slice(1, -1)
|
||||
: raw === 'true' || raw === 'false' ? raw === 'true' : Number(raw)
|
||||
}
|
||||
committed[cellId] = cell
|
||||
}
|
||||
// Each wave is pinned on its own: C30-C33 launch on the director's digest, not C27's.
|
||||
for (const wave of [
|
||||
'production-gce-c27,production-gce-c28,production-gce-c29',
|
||||
'production-gce-c30',
|
||||
'production-gce-c31',
|
||||
'production-gce-c32,production-gce-c33',
|
||||
'production-gce-c34',
|
||||
'production-gce-c35'
|
||||
]) {
|
||||
const committedImage = committed[wave.split(',')[0]].image
|
||||
assert.doesNotThrow(() => prepareRelayAsiaTopologyInput({
|
||||
existingCells: committed, existingAdditionalRegions: additionalRegions,
|
||||
environment: 'production', cellIds: wave, image: committedImage
|
||||
}), wave)
|
||||
}
|
||||
assert.throws(() => prepareRelayAsiaTopologyInput({
|
||||
existingCells: committed, existingAdditionalRegions: additionalRegions,
|
||||
environment: 'production', cellIds: 'production-gce-c30',
|
||||
image
|
||||
}), /differs from the reviewed topology/)
|
||||
// The US cells and the C34 spare launch on the newest cell digest, the one C31 launched on.
|
||||
for (const cellId of ['production-gce-c32', 'production-gce-c33', 'production-gce-c34']) {
|
||||
assert.equal(committed[cellId].image, committed['production-gce-c31'].image, cellId)
|
||||
}
|
||||
})
|
||||
|
||||
test('accepts the one exact committed staging Asia cell', () => {
|
||||
const stagingImage = image.replace('onorca-cloud/', 'onorca-cloud-staging/')
|
||||
const stagingCell = {
|
||||
hostname: 'c4', region: 'asia-east2', zone: 'asia-east2-a',
|
||||
machine_type: 'e2-standard-4', boot_disk_gb: 30,
|
||||
boot_image: 'https://www.googleapis.com/compute/v1/projects/cos-cloud/global/images/cos-121-18867-638-13',
|
||||
capacity_requests: 6_000, database_pool_max: 10, image: stagingImage,
|
||||
initially_enabled: false, connection_hard_cap: 3_000,
|
||||
connection_unobserved_bound: 60
|
||||
}
|
||||
const result = prepareRelayAsiaTopologyInput({
|
||||
existingCells: { 'staging-gce-c3': { hostname: 'c3' }, 'staging-gce-c4': stagingCell },
|
||||
existingAdditionalRegions: additionalRegions,
|
||||
environment: 'staging',
|
||||
cellIds: 'staging-gce-c4',
|
||||
image: stagingImage
|
||||
})
|
||||
assert.equal(result.relay_gce_cells['staging-gce-c4'].zone, 'asia-east2-a')
|
||||
assert.equal(result.relay_gce_cells['staging-gce-c4'].image, stagingImage)
|
||||
})
|
||||
|
||||
test('rejects an uncommitted subnet or cell, partial wave, wrong image, and drift', () => {
|
||||
for (const cellIds of [
|
||||
'production-gce-c27',
|
||||
'production-gce-c27,production-gce-c30',
|
||||
'production-gce-c27,production-gce-c28,production-gce-c29,production-gce-c30',
|
||||
'production-gce-c30,production-gce-c30',
|
||||
'production-gce-c30,production-gce-c31',
|
||||
'production-gce-c32',
|
||||
'production-gce-c33',
|
||||
'production-gce-c33,production-gce-c34',
|
||||
'production-gce-c34,production-gce-c35',
|
||||
'production-gce-c36'
|
||||
]) {
|
||||
assert.throws(() => prepareRelayAsiaTopologyInput({
|
||||
existingCells: productionCells(), existingAdditionalRegions: additionalRegions,
|
||||
environment: 'production', cellIds, image
|
||||
}), /cell IDs/, cellIds)
|
||||
}
|
||||
const poolDrift = productionCells()
|
||||
poolDrift['production-gce-c30'].database_pool_max = 10
|
||||
assert.throws(() => prepareRelayAsiaTopologyInput({
|
||||
existingCells: poolDrift, existingAdditionalRegions: additionalRegions,
|
||||
environment: 'production', cellIds: 'production-gce-c30', image
|
||||
}), /differs from the reviewed topology/)
|
||||
assert.throws(() => prepareRelayAsiaTopologyInput({
|
||||
existingCells: productionCells(), existingAdditionalRegions: additionalRegions,
|
||||
environment: 'production',
|
||||
cellIds: 'production-gce-c27,production-gce-c28,production-gce-c29',
|
||||
image: image.replace('onorca-cloud/', 'other-project/')
|
||||
}), /environment Relay image/)
|
||||
assert.throws(() => prepareRelayAsiaTopologyInput({
|
||||
existingCells: productionCells(), existingAdditionalRegions: {}, environment: 'production',
|
||||
cellIds: 'production-gce-c27,production-gce-c28,production-gce-c29', image
|
||||
}), /subnet must be committed/)
|
||||
const missing = productionCells()
|
||||
delete missing['production-gce-c29']
|
||||
assert.throws(() => prepareRelayAsiaTopologyInput({
|
||||
existingCells: missing, existingAdditionalRegions: additionalRegions, environment: 'production',
|
||||
cellIds: 'production-gce-c27,production-gce-c28,production-gce-c29', image
|
||||
}), /cells must be committed/)
|
||||
assert.throws(() => prepareRelayAsiaTopologyInput({
|
||||
existingCells: { ...productionCells(), 'production-gce-c27': {} },
|
||||
existingAdditionalRegions: additionalRegions, environment: 'production',
|
||||
cellIds: 'production-gce-c27,production-gce-c28,production-gce-c29', image
|
||||
}), /differs from the reviewed topology/)
|
||||
})
|
||||
@@ -1,183 +0,0 @@
|
||||
import { pathToFileURL } from 'node:url'
|
||||
import {
|
||||
applyExactAdmissionSelector,
|
||||
inspectAdmissionSelector,
|
||||
membershipWithStates,
|
||||
selectorCellState
|
||||
} from './relay-admission-selector.mjs'
|
||||
|
||||
function parseArguments(argv) {
|
||||
const values = {}
|
||||
for (let index = 0; index < argv.length; index += 2) {
|
||||
const key = argv[index]
|
||||
const value = argv[index + 1]
|
||||
if (!key?.startsWith('--') || value === undefined) throw new Error('invalid arguments')
|
||||
values[key.slice(2)] = value
|
||||
}
|
||||
for (const key of ['director-origin', 'cell-id', 'mode']) {
|
||||
if (!values[key]) throw new Error(`missing --${key}`)
|
||||
}
|
||||
const origin = new URL(values['director-origin'])
|
||||
if (origin.protocol !== 'https:' || origin.origin !== values['director-origin']) {
|
||||
throw new Error('--director-origin must be a canonical HTTPS origin')
|
||||
}
|
||||
if (!['isolate', 'activate', 'restore-fallback', 'restore'].includes(values.mode)) {
|
||||
throw new Error('--mode must be isolate, activate, restore-fallback, or restore')
|
||||
}
|
||||
const cellOrigin = values['cell-origin'] ? new URL(values['cell-origin']) : null
|
||||
if (
|
||||
values.mode === 'isolate' &&
|
||||
(!cellOrigin || cellOrigin.protocol !== 'https:' || cellOrigin.origin !== values['cell-origin'])
|
||||
) {
|
||||
throw new Error('--cell-origin must be a canonical HTTPS origin for isolate mode')
|
||||
}
|
||||
const restoreGeneralCellIds = values['general-cell-ids']?.split(',').filter(Boolean) ?? []
|
||||
if (
|
||||
['restore-fallback', 'restore'].includes(values.mode) &&
|
||||
restoreGeneralCellIds.length === 0
|
||||
) {
|
||||
throw new Error('--general-cell-ids is required for restore modes')
|
||||
}
|
||||
if (values.mode === 'restore' && !restoreGeneralCellIds.includes(values['cell-id'])) {
|
||||
throw new Error('--general-cell-ids must include the canary for restore mode')
|
||||
}
|
||||
if (values.mode === 'restore-fallback' && restoreGeneralCellIds.includes(values['cell-id'])) {
|
||||
throw new Error('--general-cell-ids cannot include the canary for fallback restore')
|
||||
}
|
||||
if (new Set(restoreGeneralCellIds).size !== restoreGeneralCellIds.length) {
|
||||
throw new Error('--general-cell-ids must be distinct')
|
||||
}
|
||||
return {
|
||||
directorOrigin: origin.origin,
|
||||
cellOrigin: cellOrigin?.origin,
|
||||
cellId: values['cell-id'],
|
||||
mode: values.mode,
|
||||
restoreGeneralCellIds
|
||||
}
|
||||
}
|
||||
|
||||
async function responseJson(response, label) {
|
||||
const body = await response.json().catch(() => ({}))
|
||||
if (!response.ok) throw new Error(`${label} returned ${response.status}`)
|
||||
return body
|
||||
}
|
||||
|
||||
function sameMembership(left, right) {
|
||||
return JSON.stringify(left) === JSON.stringify(right)
|
||||
}
|
||||
|
||||
async function legacyCellState(post, cellId) {
|
||||
const result = await post('/v1/admin/cell-status', { v: 1, cellId })
|
||||
if (result.status?.cellId !== cellId) throw new Error('legacy admission status is invalid')
|
||||
const state = result.status.admissionState
|
||||
if (!['existing-only', 'migration-only', 'general'].includes(state)) {
|
||||
throw new Error('legacy admission status is invalid')
|
||||
}
|
||||
return state
|
||||
}
|
||||
|
||||
async function applyLegacyStates(post, before, states, order) {
|
||||
const expected = membershipWithStates(before.selector, states)
|
||||
let changed = false
|
||||
for (const cellId of order) {
|
||||
const desired = states[cellId]
|
||||
const current = await legacyCellState(post, cellId)
|
||||
if (current === 'existing-only' && desired !== current) {
|
||||
throw new Error(`legacy admission cannot re-enable existing-only cell ${cellId}`)
|
||||
}
|
||||
if (current === desired) continue
|
||||
let cause
|
||||
try {
|
||||
await post('/v1/admin/cell-state', { v: 1, cellId, state: desired })
|
||||
} catch (error) {
|
||||
cause = error
|
||||
}
|
||||
if ((await legacyCellState(post, cellId)) !== desired) {
|
||||
const detail = cause instanceof Error ? `: ${cause.message}` : ''
|
||||
throw new Error(`legacy admission did not commit ${cellId} exactly${detail}`, { cause })
|
||||
}
|
||||
changed = true
|
||||
}
|
||||
const verified = await inspectAdmissionSelector(post)
|
||||
if (verified.selector.generation !== 0 ||
|
||||
!sameMembership(verified.selector.membership, expected)) {
|
||||
throw new Error('legacy admission membership changed unexpectedly')
|
||||
}
|
||||
return { changed, selector: verified.selector }
|
||||
}
|
||||
|
||||
export async function prepareCapacityCanary(config, overrides = {}) {
|
||||
const fetchImpl = overrides.fetch ?? fetch
|
||||
const token = overrides.token ?? process.env.ORCA_RELAY_ADMIN_ID_TOKEN
|
||||
if (!token || token.length > 8_192) throw new Error('admin identity token is unavailable')
|
||||
const postAt = async (origin, path, body) =>
|
||||
await responseJson(
|
||||
await fetchImpl(`${origin}${path}`, {
|
||||
method: 'POST',
|
||||
headers: { authorization: `Bearer ${token}`, 'content-type': 'application/json' },
|
||||
body: JSON.stringify(body),
|
||||
signal: AbortSignal.timeout(30_000)
|
||||
}),
|
||||
path
|
||||
)
|
||||
const post = async (path, body) => await postAt(config.directorOrigin, path, body)
|
||||
const before = await inspectAdmissionSelector(post)
|
||||
const state = selectorCellState(before.selector, config.cellId)
|
||||
if (state === 'existing-only' && config.mode !== 'restore-fallback') {
|
||||
throw new Error('capacity canary cannot restore existing-only admission')
|
||||
}
|
||||
const states =
|
||||
config.mode === 'isolate'
|
||||
? { [config.cellId]: 'migration-only' }
|
||||
: config.mode === 'activate'
|
||||
? Object.fromEntries([
|
||||
...before.selector.membership.general.map((cellId) => [
|
||||
cellId,
|
||||
cellId === config.cellId ? 'general' : 'migration-only'
|
||||
]),
|
||||
[config.cellId, 'general']
|
||||
])
|
||||
: Object.fromEntries([
|
||||
...config.restoreGeneralCellIds.map((cellId) => [cellId, 'general']),
|
||||
...(config.mode === 'restore-fallback'
|
||||
? [[config.cellId, state === 'existing-only' ? 'existing-only' : 'migration-only']]
|
||||
: [])
|
||||
])
|
||||
const membership = membershipWithStates(before.selector, states)
|
||||
const legacyOrder =
|
||||
config.mode === 'activate'
|
||||
? [config.cellId, ...before.selector.membership.general.filter((id) => id !== config.cellId)]
|
||||
: config.mode === 'restore-fallback'
|
||||
? [...config.restoreGeneralCellIds, config.cellId]
|
||||
: Object.keys(states)
|
||||
const result = before.selector.generation === 0
|
||||
? await applyLegacyStates(post, before, states, legacyOrder)
|
||||
: sameMembership(membership, before.selector.membership)
|
||||
? { changed: false, selector: before.selector }
|
||||
: await applyExactAdmissionSelector(post, membership, {
|
||||
expectedCurrentSelector: before.selector
|
||||
})
|
||||
if (config.mode === 'isolate') {
|
||||
await postAt(config.cellOrigin, '/v1/admin/drain', { v: 1, graceMs: 0 })
|
||||
}
|
||||
return {
|
||||
changed: result.changed,
|
||||
generation: result.selector.generation,
|
||||
...(config.mode === 'isolate' ? { drained: true } : {})
|
||||
}
|
||||
}
|
||||
|
||||
export async function main(argv = process.argv.slice(2)) {
|
||||
const config = parseArguments(argv)
|
||||
const result = await prepareCapacityCanary(config)
|
||||
process.stdout.write(
|
||||
`${JSON.stringify({ event: 'relay_capacity_canary_admission', cellId: config.cellId, mode: config.mode, ...result })}\n`
|
||||
)
|
||||
}
|
||||
|
||||
if (import.meta.url === pathToFileURL(process.argv[1]).href) {
|
||||
main().catch((error) => {
|
||||
process.stderr.write(`${error instanceof Error ? error.message : String(error)}\n`)
|
||||
process.exitCode = 1
|
||||
})
|
||||
}
|
||||
@@ -1,300 +0,0 @@
|
||||
import assert from 'node:assert/strict'
|
||||
import { readFileSync } from 'node:fs'
|
||||
import { test } from 'node:test'
|
||||
import { relayWorkflowUrl } from './relay-repository.mjs'
|
||||
import { prepareCapacityCanary } from './prepare-relay-capacity-canary.mjs'
|
||||
|
||||
function harness(initialState, options = {}) {
|
||||
const {
|
||||
generation = 4,
|
||||
ambiguousCellState = false,
|
||||
rejectCellState = false,
|
||||
fallbackState = 'general',
|
||||
extraGeneralCellIds = []
|
||||
} = options
|
||||
let selector = {
|
||||
generation,
|
||||
attemptId: 'initial',
|
||||
membership: {
|
||||
existingOnly: [
|
||||
'staging-gce-c1',
|
||||
...(initialState === 'existing-only' ? ['staging-gce-c3'] : [])
|
||||
],
|
||||
migrationOnly: [
|
||||
...(fallbackState === 'migration-only' ? ['staging-gce-c2'] : []),
|
||||
...(initialState === 'migration-only' ? ['staging-gce-c3'] : [])
|
||||
],
|
||||
general: [
|
||||
...extraGeneralCellIds,
|
||||
...(fallbackState === 'general' ? ['staging-gce-c2'] : []),
|
||||
...(initialState === 'general' ? ['staging-gce-c3'] : [])
|
||||
].sort()
|
||||
}
|
||||
}
|
||||
let intent = null
|
||||
let applies = 0
|
||||
let drains = 0
|
||||
const cellStateChanges = []
|
||||
const fetch = async (url, options) => {
|
||||
const path = new URL(url).pathname
|
||||
const body = JSON.parse(options.body)
|
||||
if (path === '/v1/admin/drain') {
|
||||
assert.deepEqual(body, { v: 1, graceMs: 0 })
|
||||
drains++
|
||||
return Response.json({ ok: true })
|
||||
}
|
||||
if (path === '/v1/admin/cell-status') {
|
||||
const state = selector.membership.existingOnly.includes(body.cellId)
|
||||
? 'existing-only'
|
||||
: selector.membership.migrationOnly.includes(body.cellId)
|
||||
? 'migration-only'
|
||||
: 'general'
|
||||
return Response.json({ status: { cellId: body.cellId, admissionState: state } })
|
||||
}
|
||||
if (path === '/v1/admin/cell-state') {
|
||||
assert.equal(selector.generation, 0)
|
||||
if (rejectCellState) {
|
||||
return Response.json({ error: 'invalid_token' }, { status: 401 })
|
||||
}
|
||||
const keys = {
|
||||
'existing-only': 'existingOnly',
|
||||
'migration-only': 'migrationOnly',
|
||||
general: 'general'
|
||||
}
|
||||
for (const cells of Object.values(selector.membership)) {
|
||||
const index = cells.indexOf(body.cellId)
|
||||
if (index !== -1) cells.splice(index, 1)
|
||||
}
|
||||
selector.membership[keys[body.state]].push(body.cellId)
|
||||
for (const cells of Object.values(selector.membership)) cells.sort()
|
||||
cellStateChanges.push({ cellId: body.cellId, state: body.state })
|
||||
if (ambiguousCellState) throw new Error('response lost')
|
||||
return Response.json({ ok: true })
|
||||
}
|
||||
if (path.endsWith('/status')) return Response.json({ selector, intent })
|
||||
applies++
|
||||
selector = {
|
||||
generation: body.expectedGeneration + 1,
|
||||
attemptId: body.attemptId,
|
||||
membership: body.membership
|
||||
}
|
||||
intent = {
|
||||
attemptId: body.attemptId,
|
||||
expectedGeneration: body.expectedGeneration,
|
||||
intendedGeneration: selector.generation,
|
||||
membership: selector.membership,
|
||||
state: 'committed'
|
||||
}
|
||||
return Response.json({ changed: true, selector })
|
||||
}
|
||||
return {
|
||||
fetch,
|
||||
selector: () => selector,
|
||||
applies: () => applies,
|
||||
drains: () => drains,
|
||||
cellStateChanges
|
||||
}
|
||||
}
|
||||
|
||||
const config = {
|
||||
directorOrigin: 'https://relay.example.com',
|
||||
cellOrigin: 'https://c3.relay.example.com',
|
||||
cellId: 'staging-gce-c3',
|
||||
mode: 'isolate',
|
||||
restoreGeneralCellIds: []
|
||||
}
|
||||
|
||||
test('isolates a general canary as migration-only', async () => {
|
||||
const testHarness = harness('general')
|
||||
assert.deepEqual(
|
||||
await prepareCapacityCanary(config, { fetch: testHarness.fetch, token: 'masked' }),
|
||||
{ changed: true, generation: 5, drained: true }
|
||||
)
|
||||
assert.deepEqual(testHarness.selector().membership.migrationOnly, [config.cellId])
|
||||
assert.equal(testHarness.applies(), 1)
|
||||
assert.equal(testHarness.drains(), 1)
|
||||
})
|
||||
|
||||
test('activates the canary as the only general cell', async () => {
|
||||
const testHarness = harness('migration-only')
|
||||
assert.deepEqual(
|
||||
await prepareCapacityCanary(
|
||||
{ ...config, mode: 'activate' },
|
||||
{ fetch: testHarness.fetch, token: 'masked' }
|
||||
),
|
||||
{ changed: true, generation: 5 }
|
||||
)
|
||||
assert.deepEqual(testHarness.selector().membership, {
|
||||
existingOnly: ['staging-gce-c1'],
|
||||
migrationOnly: ['staging-gce-c2'],
|
||||
general: [config.cellId]
|
||||
})
|
||||
assert.equal(testHarness.applies(), 1)
|
||||
})
|
||||
|
||||
test('restores the reviewed staging general membership', async () => {
|
||||
const testHarness = harness('migration-only')
|
||||
assert.deepEqual(
|
||||
await prepareCapacityCanary(
|
||||
{
|
||||
...config,
|
||||
mode: 'restore',
|
||||
restoreGeneralCellIds: ['staging-gce-c2', config.cellId]
|
||||
},
|
||||
{ fetch: testHarness.fetch, token: 'masked' }
|
||||
),
|
||||
{ changed: true, generation: 5 }
|
||||
)
|
||||
assert.deepEqual(testHarness.selector().membership, {
|
||||
existingOnly: ['staging-gce-c1'],
|
||||
migrationOnly: [],
|
||||
general: ['staging-gce-c2', config.cellId]
|
||||
})
|
||||
})
|
||||
|
||||
test('restores the fallback without promoting a possibly drained canary', async () => {
|
||||
const testHarness = harness('general')
|
||||
await prepareCapacityCanary(
|
||||
{
|
||||
...config,
|
||||
mode: 'restore-fallback',
|
||||
restoreGeneralCellIds: ['staging-gce-c2']
|
||||
},
|
||||
{ fetch: testHarness.fetch, token: 'masked' }
|
||||
)
|
||||
assert.deepEqual(testHarness.selector().membership, {
|
||||
existingOnly: ['staging-gce-c1'],
|
||||
migrationOnly: [config.cellId],
|
||||
general: ['staging-gce-c2']
|
||||
})
|
||||
})
|
||||
|
||||
test('restores the fallback while preserving an irreversible canary', async () => {
|
||||
const testHarness = harness('existing-only')
|
||||
await prepareCapacityCanary(
|
||||
{
|
||||
...config,
|
||||
mode: 'restore-fallback',
|
||||
restoreGeneralCellIds: ['staging-gce-c2']
|
||||
},
|
||||
{ fetch: testHarness.fetch, token: 'masked' }
|
||||
)
|
||||
assert.deepEqual(testHarness.selector().membership, {
|
||||
existingOnly: ['staging-gce-c1', config.cellId],
|
||||
migrationOnly: [],
|
||||
general: ['staging-gce-c2']
|
||||
})
|
||||
})
|
||||
|
||||
test('uses exact legacy admission writes before the selector boundary', async () => {
|
||||
const testHarness = harness('general', { generation: 0 })
|
||||
assert.deepEqual(
|
||||
await prepareCapacityCanary(config, { fetch: testHarness.fetch, token: 'masked' }),
|
||||
{ changed: true, generation: 0, drained: true }
|
||||
)
|
||||
assert.deepEqual(testHarness.cellStateChanges, [
|
||||
{ cellId: config.cellId, state: 'migration-only' }
|
||||
])
|
||||
assert.equal(testHarness.drains(), 1)
|
||||
})
|
||||
|
||||
test('promotes a legacy canary before demoting its fallback', async () => {
|
||||
const testHarness = harness('migration-only', { generation: 0 })
|
||||
await prepareCapacityCanary(
|
||||
{ ...config, mode: 'activate' },
|
||||
{ fetch: testHarness.fetch, token: 'masked' }
|
||||
)
|
||||
assert.deepEqual(testHarness.cellStateChanges, [
|
||||
{ cellId: config.cellId, state: 'general' },
|
||||
{ cellId: 'staging-gce-c2', state: 'migration-only' }
|
||||
])
|
||||
})
|
||||
|
||||
test('makes the canary sole general with the live legacy membership shape', async () => {
|
||||
const extraGeneralCellIds = ['combined', 'staging-c1', 'staging-c2']
|
||||
const testHarness = harness('migration-only', { generation: 0, extraGeneralCellIds })
|
||||
const activate = { ...config, mode: 'activate' }
|
||||
await prepareCapacityCanary(activate, { fetch: testHarness.fetch, token: 'masked' })
|
||||
assert.deepEqual(testHarness.cellStateChanges, [
|
||||
{ cellId: config.cellId, state: 'general' },
|
||||
{ cellId: 'combined', state: 'migration-only' },
|
||||
{ cellId: 'staging-c1', state: 'migration-only' },
|
||||
{ cellId: 'staging-c2', state: 'migration-only' },
|
||||
{ cellId: 'staging-gce-c2', state: 'migration-only' }
|
||||
])
|
||||
})
|
||||
|
||||
test('restores a legacy fallback before demoting the target', async () => {
|
||||
const testHarness = harness('general', {
|
||||
generation: 0,
|
||||
fallbackState: 'migration-only'
|
||||
})
|
||||
await prepareCapacityCanary(
|
||||
{
|
||||
...config,
|
||||
mode: 'restore-fallback',
|
||||
restoreGeneralCellIds: ['staging-gce-c2']
|
||||
},
|
||||
{ fetch: testHarness.fetch, token: 'masked' }
|
||||
)
|
||||
assert.deepEqual(testHarness.cellStateChanges, [
|
||||
{ cellId: 'staging-gce-c2', state: 'general' },
|
||||
{ cellId: config.cellId, state: 'migration-only' }
|
||||
])
|
||||
})
|
||||
|
||||
test('keeps an already restored legacy fallback unchanged', async () => {
|
||||
const testHarness = harness('migration-only', { generation: 0 })
|
||||
assert.deepEqual(
|
||||
await prepareCapacityCanary(
|
||||
{
|
||||
...config,
|
||||
mode: 'restore-fallback',
|
||||
restoreGeneralCellIds: ['staging-gce-c2']
|
||||
},
|
||||
{ fetch: testHarness.fetch, token: 'masked' }
|
||||
),
|
||||
{ changed: false, generation: 0 }
|
||||
)
|
||||
assert.deepEqual(testHarness.cellStateChanges, [])
|
||||
assert.deepEqual(testHarness.selector().membership, {
|
||||
existingOnly: ['staging-gce-c1'],
|
||||
migrationOnly: [config.cellId],
|
||||
general: ['staging-gce-c2']
|
||||
})
|
||||
})
|
||||
|
||||
test('recovers an ambiguous legacy admission response by exact readback', async () => {
|
||||
const testHarness = harness('general', { generation: 0, ambiguousCellState: true })
|
||||
await assert.doesNotReject(
|
||||
prepareCapacityCanary(config, { fetch: testHarness.fetch, token: 'masked' })
|
||||
)
|
||||
assert.equal(testHarness.drains(), 1)
|
||||
})
|
||||
|
||||
test('reports a rejected legacy admission write without draining', async () => {
|
||||
const testHarness = harness('general', { generation: 0, rejectCellState: true })
|
||||
const operation = prepareCapacityCanary(config, { fetch: testHarness.fetch, token: 'masked' })
|
||||
await assert.rejects(operation, /cell-state returned 401/)
|
||||
assert.equal(testHarness.drains(), 0)
|
||||
})
|
||||
|
||||
test('the staging workflow supplies every required capacity transition argument', () => {
|
||||
const workflow = readFileSync(
|
||||
relayWorkflowUrl('prove-relay-staging-capacity.yml'),
|
||||
'utf8'
|
||||
)
|
||||
const verifyCalls = workflow.match(
|
||||
/node dev\/scripts\/verify-relay-capacity-transition\.mjs[\s\S]*?(?=\n\s*\n|\n\s*- name:)/g
|
||||
)
|
||||
assert.ok(verifyCalls?.length >= 5)
|
||||
for (const call of verifyCalls) {
|
||||
for (const flag of ['--cell-origin', '--heartbeat', '--admission', '--draining', '--activity']) {
|
||||
assert.match(call, new RegExp(flag))
|
||||
}
|
||||
}
|
||||
const isolate = workflow.match(
|
||||
/node dev\/scripts\/prepare-relay-capacity-canary\.mjs[\s\S]*?--mode isolate/
|
||||
)?.[0]
|
||||
assert.match(isolate, /--cell-origin/)
|
||||
})
|
||||
@@ -1,196 +0,0 @@
|
||||
import { pathToFileURL } from 'node:url'
|
||||
import { fetchAdminOnceMore } from './relay-admin-transient-retry.mjs'
|
||||
import {
|
||||
applyExactAdmissionSelector,
|
||||
inspectAdmissionSelector,
|
||||
membershipWithStates,
|
||||
selectorCellState
|
||||
} from './relay-admission-selector.mjs'
|
||||
import { SAME_CAP_CELLS } from './relay-production-same-cap-wave.mjs'
|
||||
|
||||
const DIRECTOR_ORIGIN = 'https://relay.onorca.dev'
|
||||
export const PRODUCTION_CAPACITY_CELL_IDS = [
|
||||
'production-gce-c7',
|
||||
'production-gce-c8',
|
||||
'production-gce-c9',
|
||||
'production-gce-c10',
|
||||
'production-gce-c13',
|
||||
'production-gce-c14',
|
||||
'production-gce-c15',
|
||||
'production-gce-c16',
|
||||
'production-gce-c19',
|
||||
'production-gce-c20',
|
||||
'production-gce-c21',
|
||||
'production-gce-c22',
|
||||
'production-gce-c23',
|
||||
'production-gce-c24',
|
||||
'production-gce-c25',
|
||||
'production-gce-c26'
|
||||
]
|
||||
|
||||
function cellOrigin(cellId) {
|
||||
return `https://${cellId.slice('production-gce-'.length)}.relay.onorca.dev`
|
||||
}
|
||||
|
||||
// The same-cap roll covers the Asia cells the US-only capacity rollout never touches.
|
||||
const APPROVED_CELL_LISTS = { 'same-cap': SAME_CAP_CELLS }
|
||||
|
||||
// Matches the cell's own cap on /v1/admin/drain.
|
||||
const MAX_PACE_WINDOW_MS = 20 * 60 * 1_000
|
||||
// Every paced image before the cap rose rejects a longer window with the same 400 an unpaced
|
||||
// image gives, and an unpaced drain is the opposite of what a slower window asked for.
|
||||
const LEGACY_MAX_PACE_WINDOW_MS = 5 * 60 * 1_000
|
||||
|
||||
export function parseProductionCapacityCellArguments(argv) {
|
||||
const values = {}
|
||||
for (let index = 0; index < argv.length; index += 2) {
|
||||
const key = argv[index]
|
||||
const value = argv[index + 1]
|
||||
if (!key?.startsWith('--') || value === undefined) throw new Error('invalid arguments')
|
||||
values[key.slice(2)] = value
|
||||
}
|
||||
if (!['pace-check', 'isolate', 'drain', 'activate'].includes(values.mode)) {
|
||||
throw new Error('--mode must be pace-check, isolate, drain, or activate')
|
||||
}
|
||||
const approvedList = values['approved-cells']
|
||||
if (approvedList !== undefined && !APPROVED_CELL_LISTS[approvedList]) {
|
||||
throw new Error('--approved-cells is not a known allowlist')
|
||||
}
|
||||
const approvedCellIds = approvedList === undefined
|
||||
? PRODUCTION_CAPACITY_CELL_IDS
|
||||
: APPROVED_CELL_LISTS[approvedList]
|
||||
const cellId = values['cell-id']
|
||||
if (!approvedCellIds.includes(cellId)) {
|
||||
throw new Error('production capacity target is not approved')
|
||||
}
|
||||
const expectedCellOrigin = cellOrigin(cellId)
|
||||
if (
|
||||
values['director-origin'] !== DIRECTOR_ORIGIN ||
|
||||
values['cell-origin'] !== expectedCellOrigin
|
||||
) {
|
||||
throw new Error('production capacity target origin is not exact')
|
||||
}
|
||||
const paceWindowMs = values['pace-window-ms'] === undefined
|
||||
? 0
|
||||
: Number(values['pace-window-ms'])
|
||||
if (
|
||||
!Number.isSafeInteger(paceWindowMs) ||
|
||||
paceWindowMs < 0 ||
|
||||
paceWindowMs > MAX_PACE_WINDOW_MS
|
||||
) {
|
||||
throw new Error('--pace-window-ms must be an integer between 0 and 1200000')
|
||||
}
|
||||
return {
|
||||
directorOrigin: DIRECTOR_ORIGIN,
|
||||
cellOrigin: expectedCellOrigin,
|
||||
cellId,
|
||||
mode: values.mode,
|
||||
paceWindowMs
|
||||
}
|
||||
}
|
||||
|
||||
async function responseJson(response, label) {
|
||||
const body = await response.json().catch(() => ({}))
|
||||
if (!response.ok) throw new Error(`${label} returned ${response.status}`)
|
||||
return body
|
||||
}
|
||||
|
||||
// Read-only, and run before the isolate: an image that does not advertise its cap has the
|
||||
// legacy one, so a slower pace stops here with the cell untouched instead of isolated.
|
||||
async function checkDrainPace(fetchImpl, config, wait) {
|
||||
const health = await responseJson(
|
||||
await fetchAdminOnceMore(fetchImpl, `${config.cellOrigin}/health`, {}, { wait }),
|
||||
'cell health'
|
||||
)
|
||||
if (health.ok !== true) throw new Error('cell health is not ok')
|
||||
const maxPaceWindowMs = Number.isSafeInteger(health.drainPaceWindowMaxMs)
|
||||
? health.drainPaceWindowMaxMs
|
||||
: LEGACY_MAX_PACE_WINDOW_MS
|
||||
if (config.paceWindowMs > maxPaceWindowMs) {
|
||||
throw new Error(
|
||||
`cell accepts drain paces up to ${maxPaceWindowMs} ms, not ${config.paceWindowMs} ms; ` +
|
||||
'it was not isolated'
|
||||
)
|
||||
}
|
||||
return { paceWindowMs: config.paceWindowMs, maxPaceWindowMs }
|
||||
}
|
||||
|
||||
export async function prepareProductionCapacityCell(config, overrides = {}) {
|
||||
const fetchImpl = overrides.fetch ?? fetch
|
||||
if (config.mode === 'pace-check') return await checkDrainPace(fetchImpl, config, overrides.wait)
|
||||
const token = overrides.token ?? process.env.ORCA_RELAY_ADMIN_ID_TOKEN
|
||||
if (!token || token.length > 8_192) throw new Error('admin identity token is unavailable')
|
||||
const postRaw = async (origin, path, body) =>
|
||||
await fetchAdminOnceMore(
|
||||
fetchImpl,
|
||||
`${origin}${path}`,
|
||||
{
|
||||
method: 'POST',
|
||||
headers: { authorization: `Bearer ${token}`, 'content-type': 'application/json' },
|
||||
body: JSON.stringify(body)
|
||||
},
|
||||
{ wait: overrides.wait }
|
||||
)
|
||||
const postAt = async (origin, path, body) =>
|
||||
await responseJson(await postRaw(origin, path, body), path)
|
||||
const post = async (path, body) => await postAt(config.directorOrigin, path, body)
|
||||
if (config.mode === 'drain') {
|
||||
const paceWindowMs = config.paceWindowMs ?? 0
|
||||
if (paceWindowMs > 0) {
|
||||
const paced = await postRaw(config.cellOrigin, '/v1/admin/drain', {
|
||||
v: 1,
|
||||
graceMs: 0,
|
||||
paceWindowMs
|
||||
})
|
||||
if (paced.ok) {
|
||||
await paced.json().catch(() => ({}))
|
||||
return { changed: false, drained: true, paceWindowMs }
|
||||
}
|
||||
// A cell still on an image without paced drain rejects the unknown field outright.
|
||||
// An unpaced drain is the behaviour that cell already has, so fall back to it.
|
||||
if (paced.status !== 400) throw new Error(`/v1/admin/drain returned ${paced.status}`)
|
||||
await paced.json().catch(() => ({}))
|
||||
if (paceWindowMs > LEGACY_MAX_PACE_WINDOW_MS) {
|
||||
throw new Error(
|
||||
`cell rejected a ${paceWindowMs} ms drain pace; refusing to drain faster than asked`
|
||||
)
|
||||
}
|
||||
}
|
||||
await postAt(config.cellOrigin, '/v1/admin/drain', { v: 1, graceMs: 0 })
|
||||
return { changed: false, drained: true, paceWindowMs: 0 }
|
||||
}
|
||||
const before = await inspectAdmissionSelector(post)
|
||||
const state = selectorCellState(before.selector, config.cellId)
|
||||
if (state === 'existing-only') throw new Error('production capacity target is irreversible')
|
||||
const desiredState = config.mode === 'isolate' ? 'migration-only' : 'general'
|
||||
const membership = membershipWithStates(before.selector, { [config.cellId]: desiredState })
|
||||
const result = await applyExactAdmissionSelector(post, membership, {
|
||||
expectedCurrentSelector: before.selector,
|
||||
// The stamp that tells the director this cell is parked for a restart rather
|
||||
// than held back as capacity, so it may re-place the hosts still on it. The
|
||||
// activate branch omits it, and moving to 'general' clears it in the same
|
||||
// statement that writes the state.
|
||||
...(config.mode === 'isolate' ? { rollIsolatedCells: [config.cellId] } : {})
|
||||
})
|
||||
return {
|
||||
changed: result.changed,
|
||||
generation: result.selector.generation,
|
||||
admissionState: desiredState,
|
||||
rollIsolated: config.mode === 'isolate'
|
||||
}
|
||||
}
|
||||
|
||||
export async function main(argv = process.argv.slice(2)) {
|
||||
const config = parseProductionCapacityCellArguments(argv)
|
||||
const result = await prepareProductionCapacityCell(config)
|
||||
process.stdout.write(
|
||||
`${JSON.stringify({ event: 'relay_production_capacity_canary', cellId: config.cellId, mode: config.mode, ...result })}\n`
|
||||
)
|
||||
}
|
||||
|
||||
if (import.meta.url === pathToFileURL(process.argv[1]).href) {
|
||||
main().catch((error) => {
|
||||
process.stderr.write(`${error instanceof Error ? error.message : String(error)}\n`)
|
||||
process.exitCode = 1
|
||||
})
|
||||
}
|
||||
@@ -1,391 +0,0 @@
|
||||
import assert from 'node:assert/strict'
|
||||
import { describe, it } from 'node:test'
|
||||
import {
|
||||
parseProductionCapacityCellArguments,
|
||||
prepareProductionCapacityCell
|
||||
} from './prepare-relay-production-capacity-canary.mjs'
|
||||
|
||||
const config = {
|
||||
directorOrigin: 'https://relay.onorca.dev',
|
||||
cellOrigin: 'https://c26.relay.onorca.dev',
|
||||
cellId: 'production-gce-c26'
|
||||
}
|
||||
|
||||
const membership = {
|
||||
existingOnly: ['production-gce-c1'],
|
||||
migrationOnly: ['production-gce-c17'],
|
||||
general: ['production-gce-c25', 'production-gce-c26']
|
||||
}
|
||||
|
||||
function response(body, status = 200) {
|
||||
return new Response(JSON.stringify(body), {
|
||||
status,
|
||||
headers: { 'content-type': 'application/json' }
|
||||
})
|
||||
}
|
||||
|
||||
function canaryFetch() {
|
||||
let selector = { generation: 20, attemptId: null, membership }
|
||||
const calls = []
|
||||
const fetch = async (url, init) => {
|
||||
const path = new URL(url).pathname
|
||||
const body = JSON.parse(init.body)
|
||||
calls.push({ path, body })
|
||||
if (path === '/v1/admin/admission-selector/status') {
|
||||
return response({
|
||||
v: 1,
|
||||
selector,
|
||||
intent: body.attemptId
|
||||
? {
|
||||
attemptId: body.attemptId,
|
||||
state: 'committed',
|
||||
expectedGeneration: selector.generation - 1,
|
||||
intendedGeneration: selector.generation,
|
||||
membership: selector.membership
|
||||
}
|
||||
: null
|
||||
})
|
||||
}
|
||||
if (path === '/v1/admin/admission-selector/apply') {
|
||||
selector = {
|
||||
generation: selector.generation + 1,
|
||||
attemptId: body.attemptId,
|
||||
membership: body.membership
|
||||
}
|
||||
return response({ v: 1, changed: true, selector })
|
||||
}
|
||||
if (path === '/v1/admin/drain') return response({ v: 1, draining: true })
|
||||
throw new Error(`unexpected ${path}`)
|
||||
}
|
||||
return { calls, fetch, selector: () => selector }
|
||||
}
|
||||
|
||||
describe('production Relay capacity cell admission', () => {
|
||||
it('allows only the serving rollout cells', () => {
|
||||
assert.deepEqual(parseProductionCapacityCellArguments([
|
||||
'--director-origin', 'https://relay.onorca.dev',
|
||||
'--cell-origin', 'https://c7.relay.onorca.dev',
|
||||
'--cell-id', 'production-gce-c7',
|
||||
'--mode', 'isolate'
|
||||
]), {
|
||||
directorOrigin: 'https://relay.onorca.dev',
|
||||
cellOrigin: 'https://c7.relay.onorca.dev',
|
||||
cellId: 'production-gce-c7',
|
||||
mode: 'isolate',
|
||||
paceWindowMs: 0
|
||||
})
|
||||
assert.throws(() => parseProductionCapacityCellArguments([
|
||||
'--director-origin', 'https://relay.onorca.dev',
|
||||
'--cell-origin', 'https://c17.relay.onorca.dev',
|
||||
'--cell-id', 'production-gce-c17',
|
||||
'--mode', 'isolate'
|
||||
]), /not approved/)
|
||||
assert.throws(() => parseProductionCapacityCellArguments([
|
||||
'--director-origin', 'https://relay.onorca.dev',
|
||||
'--cell-origin', 'https://c8.relay.onorca.dev',
|
||||
'--cell-id', 'production-gce-c7',
|
||||
'--mode', 'isolate'
|
||||
]), /origin is not exact/)
|
||||
assert.throws(() => parseProductionCapacityCellArguments([
|
||||
'--director-origin', 'https://relay.onorca.dev',
|
||||
'--cell-origin', 'https://c27.relay.onorca.dev',
|
||||
'--cell-id', 'production-gce-c27',
|
||||
'--mode', 'isolate'
|
||||
]), /not approved/)
|
||||
})
|
||||
|
||||
it('admits the same-cap Asia and migration-only cells only under the same-cap allowlist', () => {
|
||||
for (const cellId of [
|
||||
'production-gce-c27', 'production-gce-c28', 'production-gce-c29', 'production-gce-c30',
|
||||
'production-gce-c31', 'production-gce-c34', 'production-gce-c35',
|
||||
// Migration-only canaries: the US-only capacity rollout never touches them either.
|
||||
'production-gce-c17', 'production-gce-c18',
|
||||
// US cells at the Asia shape: the 1,000-cap capacity rollout never touches them.
|
||||
'production-gce-c32', 'production-gce-c33'
|
||||
]) {
|
||||
const hostname = cellId.slice('production-gce-'.length)
|
||||
assert.deepEqual(parseProductionCapacityCellArguments([
|
||||
'--director-origin', 'https://relay.onorca.dev',
|
||||
'--cell-origin', `https://${hostname}.relay.onorca.dev`,
|
||||
'--cell-id', cellId,
|
||||
'--approved-cells', 'same-cap',
|
||||
'--mode', 'isolate'
|
||||
]), {
|
||||
directorOrigin: 'https://relay.onorca.dev',
|
||||
cellOrigin: `https://${hostname}.relay.onorca.dev`,
|
||||
cellId,
|
||||
mode: 'isolate',
|
||||
paceWindowMs: 0
|
||||
})
|
||||
}
|
||||
for (const cellId of ['production-gce-c12', 'production-gce-c36']) {
|
||||
const hostname = cellId.slice('production-gce-'.length)
|
||||
assert.throws(() => parseProductionCapacityCellArguments([
|
||||
'--director-origin', 'https://relay.onorca.dev',
|
||||
'--cell-origin', `https://${hostname}.relay.onorca.dev`,
|
||||
'--cell-id', cellId,
|
||||
'--approved-cells', 'same-cap',
|
||||
'--mode', 'isolate'
|
||||
]), /not approved/)
|
||||
}
|
||||
assert.throws(() => parseProductionCapacityCellArguments([
|
||||
'--director-origin', 'https://relay.onorca.dev',
|
||||
'--cell-origin', 'https://c27.relay.onorca.dev',
|
||||
'--cell-id', 'production-gce-c27',
|
||||
'--approved-cells', 'every-cell',
|
||||
'--mode', 'isolate'
|
||||
]), /not a known allowlist/)
|
||||
})
|
||||
|
||||
it('isolates only the selected cell without depending on its runtime', async () => {
|
||||
const fake = canaryFetch()
|
||||
const result = await prepareProductionCapacityCell(
|
||||
{ ...config, mode: 'isolate' },
|
||||
{ fetch: fake.fetch, token: 'token' }
|
||||
)
|
||||
assert.equal(result.admissionState, 'migration-only')
|
||||
assert.deepEqual(fake.selector().membership, {
|
||||
existingOnly: ['production-gce-c1'],
|
||||
migrationOnly: ['production-gce-c17', 'production-gce-c26'],
|
||||
general: ['production-gce-c25']
|
||||
})
|
||||
assert.doesNotMatch(fake.calls.map(({ path }) => path).join(','), /\/v1\/admin\/drain/)
|
||||
})
|
||||
|
||||
it('stamps the roll isolation on isolate and never on activate', async () => {
|
||||
// The stamp is what lets the director tell a cell parked for a restart from
|
||||
// an evacuation target or an Asia rollback, both of which must keep their
|
||||
// hosts. Only this call site may send it.
|
||||
const isolate = canaryFetch()
|
||||
await prepareProductionCapacityCell(
|
||||
{ ...config, mode: 'isolate' },
|
||||
{ fetch: isolate.fetch, token: 'token' }
|
||||
)
|
||||
const isolateApply = isolate.calls.find(
|
||||
({ path }) => path === '/v1/admin/admission-selector/apply'
|
||||
)
|
||||
assert.deepEqual(isolateApply.body.rollIsolatedCells, [config.cellId])
|
||||
|
||||
// Restore has to be a real apply, not the no-op an already-general cell
|
||||
// takes, or the assertion below proves nothing.
|
||||
isolate.calls.length = 0
|
||||
const restored = await prepareProductionCapacityCell(
|
||||
{ ...config, mode: 'activate' },
|
||||
{ fetch: isolate.fetch, token: 'token' }
|
||||
)
|
||||
assert.equal(restored.admissionState, 'general')
|
||||
const restoreApply = isolate.calls.find(
|
||||
({ path }) => path === '/v1/admin/admission-selector/apply'
|
||||
)
|
||||
assert.ok(restoreApply, 'restore must issue an apply')
|
||||
assert.equal(restoreApply.body.rollIsolatedCells, undefined)
|
||||
assert.ok(restoreApply.body.membership.general.includes(config.cellId))
|
||||
})
|
||||
|
||||
it('drains the selected cell independently after durable isolation', async () => {
|
||||
const fake = canaryFetch()
|
||||
const result = await prepareProductionCapacityCell(
|
||||
{ ...config, mode: 'drain' },
|
||||
{ fetch: fake.fetch, token: 'token' }
|
||||
)
|
||||
assert.deepEqual(result, { changed: false, drained: true, paceWindowMs: 0 })
|
||||
assert.deepEqual(fake.calls, [{
|
||||
path: '/v1/admin/drain',
|
||||
body: { v: 1, graceMs: 0 }
|
||||
}])
|
||||
})
|
||||
|
||||
it('paces the drain send when the roll asks for a window', async () => {
|
||||
const fake = canaryFetch()
|
||||
const result = await prepareProductionCapacityCell(
|
||||
{ ...config, mode: 'drain', paceWindowMs: 120_000 },
|
||||
{ fetch: fake.fetch, token: 'token' }
|
||||
)
|
||||
assert.deepEqual(result, { changed: false, drained: true, paceWindowMs: 120_000 })
|
||||
assert.deepEqual(fake.calls, [{
|
||||
path: '/v1/admin/drain',
|
||||
body: { v: 1, graceMs: 0, paceWindowMs: 120_000 }
|
||||
}])
|
||||
})
|
||||
|
||||
it('drains unpaced when the cell image rejects the pacing field', async () => {
|
||||
const bodies = []
|
||||
const result = await prepareProductionCapacityCell(
|
||||
{ ...config, mode: 'drain', paceWindowMs: 120_000 },
|
||||
{
|
||||
token: 'token',
|
||||
wait: async () => {},
|
||||
fetch: async (url, init) => {
|
||||
assert.equal(new URL(url).pathname, '/v1/admin/drain')
|
||||
const body = JSON.parse(init.body)
|
||||
bodies.push(body)
|
||||
if (body.paceWindowMs !== undefined) return response({ error: 'invalid_request' }, 400)
|
||||
return response({ v: 1, draining: true })
|
||||
}
|
||||
}
|
||||
)
|
||||
assert.deepEqual(result, { changed: false, drained: true, paceWindowMs: 0 })
|
||||
assert.deepEqual(bodies, [
|
||||
{ v: 1, graceMs: 0, paceWindowMs: 120_000 },
|
||||
{ v: 1, graceMs: 0 }
|
||||
])
|
||||
})
|
||||
|
||||
it('refuses to fall back to an unpaced drain when the cell rejects a slower window', async () => {
|
||||
for (const paceWindowMs of [900_000, 1_200_000]) {
|
||||
const bodies = []
|
||||
await assert.rejects(
|
||||
prepareProductionCapacityCell(
|
||||
{ ...config, mode: 'drain', paceWindowMs },
|
||||
{
|
||||
token: 'token',
|
||||
wait: async () => {},
|
||||
fetch: async (_url, init) => {
|
||||
bodies.push(JSON.parse(init.body))
|
||||
return response({ error: 'invalid_request' }, 400)
|
||||
}
|
||||
}
|
||||
),
|
||||
/refusing to drain faster than asked/
|
||||
)
|
||||
assert.deepEqual(bodies, [{ v: 1, graceMs: 0, paceWindowMs }])
|
||||
}
|
||||
})
|
||||
|
||||
it('stops a slower pace than an old-cap cell accepts before anything is isolated', async () => {
|
||||
const check = async (health, paceWindowMs) => {
|
||||
const requests = []
|
||||
const result = prepareProductionCapacityCell(
|
||||
{ ...config, mode: 'pace-check', paceWindowMs },
|
||||
{
|
||||
wait: async () => {},
|
||||
fetch: async (url, init) => {
|
||||
requests.push({ path: new URL(url).pathname, method: init.method ?? 'GET' })
|
||||
return response(health)
|
||||
}
|
||||
}
|
||||
)
|
||||
return { result, requests }
|
||||
}
|
||||
// Every image before the cap rose answers /health without it.
|
||||
const old = { ok: true, connectionCapacityProtocol: 2 }
|
||||
for (const pace of [900_000, 1_200_000]) {
|
||||
const { result, requests } = await check(old, pace)
|
||||
await assert.rejects(result, /up to 300000 ms, not \d+ ms; it was not isolated/)
|
||||
assert.deepEqual(requests, [{ path: '/health', method: 'GET' }])
|
||||
}
|
||||
assert.deepEqual(await (await check(old, 300_000)).result, {
|
||||
paceWindowMs: 300_000, maxPaceWindowMs: 300_000
|
||||
})
|
||||
const current = { ...old, drainPaceWindowMaxMs: 1_200_000 }
|
||||
assert.deepEqual(await (await check(current, 1_200_000)).result, {
|
||||
paceWindowMs: 1_200_000, maxPaceWindowMs: 1_200_000
|
||||
})
|
||||
await assert.rejects((await check({ ok: false }, 300_000)).result, /cell health is not ok/)
|
||||
})
|
||||
|
||||
it('fails a paced drain that the cell rejects for any other reason', async () => {
|
||||
await assert.rejects(
|
||||
prepareProductionCapacityCell(
|
||||
{ ...config, mode: 'drain', paceWindowMs: 120_000 },
|
||||
{
|
||||
token: 'token',
|
||||
wait: async () => {},
|
||||
fetch: async () => response({ error: 'invalid_token' }, 401)
|
||||
}
|
||||
),
|
||||
/returned 401/
|
||||
)
|
||||
})
|
||||
|
||||
it('refuses a pacing window that is not a bounded integer', () => {
|
||||
const argv = (value) => [
|
||||
'--director-origin', 'https://relay.onorca.dev',
|
||||
'--cell-origin', 'https://c26.relay.onorca.dev',
|
||||
'--cell-id', 'production-gce-c26',
|
||||
'--mode', 'drain',
|
||||
'--pace-window-ms', value
|
||||
]
|
||||
for (const value of ['-1', '1200001', '1.5', 'soon']) {
|
||||
assert.throws(() => parseProductionCapacityCellArguments(argv(value)), /pace-window-ms/)
|
||||
}
|
||||
for (const value of [300_000, 900_000, 1_200_000]) {
|
||||
assert.equal(parseProductionCapacityCellArguments(argv(String(value))).paceWindowMs, value)
|
||||
}
|
||||
})
|
||||
|
||||
it('restores only the selected cell to general admission', async () => {
|
||||
const fake = canaryFetch()
|
||||
await prepareProductionCapacityCell(
|
||||
{ ...config, mode: 'isolate' },
|
||||
{ fetch: fake.fetch, token: 'token' }
|
||||
)
|
||||
const result = await prepareProductionCapacityCell(
|
||||
{ ...config, mode: 'activate' },
|
||||
{ fetch: fake.fetch, token: 'token' }
|
||||
)
|
||||
assert.equal(result.admissionState, 'general')
|
||||
assert.deepEqual(fake.selector().membership, membership)
|
||||
})
|
||||
|
||||
it('refuses an irreversible existing-only target', async () => {
|
||||
const fetch = async () => response({
|
||||
v: 1,
|
||||
selector: {
|
||||
generation: 20,
|
||||
attemptId: null,
|
||||
membership: {
|
||||
existingOnly: ['production-gce-c26'],
|
||||
migrationOnly: ['production-gce-c17'],
|
||||
general: ['production-gce-c25']
|
||||
}
|
||||
},
|
||||
intent: null
|
||||
})
|
||||
await assert.rejects(
|
||||
prepareProductionCapacityCell(
|
||||
{ ...config, mode: 'isolate' },
|
||||
{ fetch, token: 'token' }
|
||||
),
|
||||
/irreversible/
|
||||
)
|
||||
})
|
||||
|
||||
it('retries a transient 503 on the cell drain endpoint', async () => {
|
||||
let calls = 0
|
||||
const result = await prepareProductionCapacityCell(
|
||||
{ ...config, mode: 'drain' },
|
||||
{
|
||||
token: 'token',
|
||||
wait: async () => {},
|
||||
fetch: async (url) => {
|
||||
assert.equal(new URL(url).pathname, '/v1/admin/drain')
|
||||
calls += 1
|
||||
if (calls === 1) return response({ error: 'warming up' }, 503)
|
||||
return response({ v: 1, draining: true })
|
||||
}
|
||||
}
|
||||
)
|
||||
assert.equal(calls, 2)
|
||||
assert.deepEqual(result, { changed: false, drained: true, paceWindowMs: 0 })
|
||||
})
|
||||
|
||||
it('fails when both drain attempts return a transient 503', async () => {
|
||||
let calls = 0
|
||||
await assert.rejects(
|
||||
prepareProductionCapacityCell(
|
||||
{ ...config, mode: 'drain' },
|
||||
{
|
||||
token: 'token',
|
||||
wait: async () => {},
|
||||
fetch: async () => {
|
||||
calls += 1
|
||||
return response({ error: 'warming up' }, 503)
|
||||
}
|
||||
}
|
||||
),
|
||||
/returned 503/
|
||||
)
|
||||
assert.equal(calls, 2)
|
||||
})
|
||||
})
|
||||
@@ -1,73 +0,0 @@
|
||||
import { randomBytes } from 'node:crypto'
|
||||
import { pathToFileURL } from 'node:url'
|
||||
|
||||
const WRONG_CELL = 4409
|
||||
const DRAINING = 4503
|
||||
|
||||
function once(socket, event, listener) {
|
||||
if (typeof socket.once === 'function') {
|
||||
socket.once(event, listener)
|
||||
return
|
||||
}
|
||||
if (typeof socket.addEventListener !== 'function') {
|
||||
throw new Error('WebSocket event API is unavailable')
|
||||
}
|
||||
socket.addEventListener(event, (value) => {
|
||||
if (event === 'close') listener(value.code)
|
||||
else if (event === 'error') listener(value.error ?? new Error(value.message))
|
||||
else listener()
|
||||
}, { once: true })
|
||||
}
|
||||
|
||||
export function parseLegacyAdmissionProbeArguments(argv) {
|
||||
if (argv.length !== 2 || argv[0] !== '--cell-origin') throw new Error('invalid arguments')
|
||||
const origin = new URL(argv[1])
|
||||
if (origin.protocol !== 'https:' || origin.origin !== argv[1]) {
|
||||
throw new Error('--cell-origin must be a canonical HTTPS origin')
|
||||
}
|
||||
return { cellOrigin: origin.origin }
|
||||
}
|
||||
|
||||
export async function probeLegacyAdmission(config, overrides = {}) {
|
||||
const Socket = overrides.WebSocket ?? globalThis.WebSocket
|
||||
if (typeof Socket !== 'function') throw new Error('WebSocket is unavailable')
|
||||
const random = overrides.randomBytes ?? randomBytes
|
||||
const timeoutMs = overrides.timeoutMs ?? 15_000
|
||||
const hostId = random(12).toString('base64url')
|
||||
const credential = random(32).toString('base64url')
|
||||
const url = `${config.cellOrigin.replace('https://', 'wss://')}/v1/connect/${hostId}`
|
||||
await new Promise((resolve, reject) => {
|
||||
const socket = new Socket(url)
|
||||
const timer = setTimeout(() => {
|
||||
if (typeof socket.terminate === 'function') socket.terminate()
|
||||
else socket.close()
|
||||
reject(new Error('legacy admission probe timed out'))
|
||||
}, timeoutMs)
|
||||
once(socket, 'open', () => {
|
||||
socket.send(JSON.stringify({ type: 'relay-auth', v: 1, mode: 'connect', credential }))
|
||||
})
|
||||
once(socket, 'close', (code) => {
|
||||
clearTimeout(timer)
|
||||
if (code === WRONG_CELL) resolve()
|
||||
else if (code === DRAINING) reject(new Error('legacy cell is draining'))
|
||||
else reject(new Error(`legacy admission probe closed with ${code}`))
|
||||
})
|
||||
once(socket, 'error', (error) => {
|
||||
clearTimeout(timer)
|
||||
reject(new Error(`legacy admission probe failed: ${error.message}`))
|
||||
})
|
||||
})
|
||||
return { accepting: true }
|
||||
}
|
||||
|
||||
export async function main(argv = process.argv.slice(2)) {
|
||||
const result = await probeLegacyAdmission(parseLegacyAdmissionProbeArguments(argv))
|
||||
process.stdout.write(`${JSON.stringify({ event: 'relay_legacy_admission_verified', ...result })}\n`)
|
||||
}
|
||||
|
||||
if (import.meta.url === pathToFileURL(process.argv[1]).href) {
|
||||
main().catch((error) => {
|
||||
process.stderr.write(`${error instanceof Error ? error.message : String(error)}\n`)
|
||||
process.exitCode = 1
|
||||
})
|
||||
}
|
||||
@@ -1,89 +0,0 @@
|
||||
import assert from 'node:assert/strict'
|
||||
import { EventEmitter } from 'node:events'
|
||||
import { test } from 'node:test'
|
||||
import {
|
||||
parseLegacyAdmissionProbeArguments,
|
||||
probeLegacyAdmission
|
||||
} from './probe-relay-legacy-admission.mjs'
|
||||
|
||||
function socketClosingWith(code, observed) {
|
||||
return class extends EventEmitter {
|
||||
constructor(url) {
|
||||
super()
|
||||
observed.url = url
|
||||
queueMicrotask(() => this.emit('open'))
|
||||
}
|
||||
|
||||
send(payload) {
|
||||
observed.payload = JSON.parse(payload)
|
||||
queueMicrotask(() => this.emit('close', code))
|
||||
}
|
||||
|
||||
terminate() {}
|
||||
}
|
||||
}
|
||||
|
||||
function nativeSocketClosingWith(code) {
|
||||
return class extends EventTarget {
|
||||
constructor() {
|
||||
super()
|
||||
queueMicrotask(() => this.dispatchEvent(new Event('open')))
|
||||
}
|
||||
|
||||
send() {
|
||||
const event = new Event('close')
|
||||
Object.defineProperty(event, 'code', { value: code })
|
||||
queueMicrotask(() => this.dispatchEvent(event))
|
||||
}
|
||||
|
||||
close() {}
|
||||
}
|
||||
}
|
||||
|
||||
const config = { cellOrigin: 'https://c2.relay.example.com' }
|
||||
const random = (length) => Buffer.alloc(length, length)
|
||||
|
||||
test('accepts only a canonical cell origin', () => {
|
||||
assert.deepEqual(
|
||||
parseLegacyAdmissionProbeArguments(['--cell-origin', config.cellOrigin]),
|
||||
config
|
||||
)
|
||||
assert.throws(
|
||||
() => parseLegacyAdmissionProbeArguments(['--cell-origin', `${config.cellOrigin}/path`]),
|
||||
/canonical/
|
||||
)
|
||||
})
|
||||
|
||||
test('proves admission with a synthetic invalid credential and exposes no identifier', async () => {
|
||||
const observed = {}
|
||||
assert.deepEqual(
|
||||
await probeLegacyAdmission(config, {
|
||||
WebSocket: socketClosingWith(4409, observed),
|
||||
randomBytes: random
|
||||
}),
|
||||
{ accepting: true }
|
||||
)
|
||||
assert.match(observed.url, /^wss:\/\/c2\.relay\.example\.com\/v1\/connect\/[A-Za-z0-9_-]{16}$/)
|
||||
assert.deepEqual(Object.keys(observed.payload).sort(), ['credential', 'mode', 'type', 'v'])
|
||||
})
|
||||
|
||||
test('uses the dependency-free Node WebSocket event API', async () => {
|
||||
await assert.doesNotReject(
|
||||
probeLegacyAdmission(config, {
|
||||
WebSocket: nativeSocketClosingWith(4409),
|
||||
randomBytes: random
|
||||
})
|
||||
)
|
||||
})
|
||||
|
||||
test('rejects the legacy draining close and any unknown outcome', async () => {
|
||||
for (const [code, message] of [[4503, /draining/], [4401, /closed with 4401/]]) {
|
||||
await assert.rejects(
|
||||
probeLegacyAdmission(config, {
|
||||
WebSocket: socketClosingWith(code, {}),
|
||||
randomBytes: random
|
||||
}),
|
||||
message
|
||||
)
|
||||
}
|
||||
})
|
||||
@@ -1,112 +0,0 @@
|
||||
import { pathToFileURL } from 'node:url'
|
||||
import { fetchAdminOnceMore } from './relay-admin-transient-retry.mjs'
|
||||
|
||||
// Every cell that carries the rehome identity: the eighteen US cells and the
|
||||
// seven asia-east2 cells that drain mis-homed hosts back the other way.
|
||||
const PRODUCTION_CELL =
|
||||
/^production-gce-c(?:7|8|9|10|13|14|15|16|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35)$/
|
||||
const DIRECTOR_ORIGIN = 'https://relay.onorca.dev'
|
||||
const SOURCE_TRANSIENT_RETRIES = 12
|
||||
const SOURCE_TRANSIENT_RETRY_DELAY_MS = 10_000
|
||||
|
||||
export function parseRehomeTrustProbeArguments(argv, environment = process.env) {
|
||||
const values = {}
|
||||
for (let index = 0; index < argv.length; index += 2) {
|
||||
const key = argv[index]
|
||||
const value = argv[index + 1]
|
||||
if (!key?.startsWith('--') || value === undefined) throw new Error('invalid arguments')
|
||||
values[key.slice(2)] = value
|
||||
}
|
||||
for (const key of ['director-origin', 'cell-id', 'cell-incarnation']) {
|
||||
if (!values[key]) throw new Error(`missing --${key}`)
|
||||
}
|
||||
if (values['director-origin'] !== DIRECTOR_ORIGIN) {
|
||||
throw new Error('--director-origin must be the production Relay origin')
|
||||
}
|
||||
if (!PRODUCTION_CELL.test(values['cell-id'])) throw new Error('--cell-id is not approved')
|
||||
if (!/^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i.test(
|
||||
values['cell-incarnation']
|
||||
)) throw new Error('--cell-incarnation is invalid')
|
||||
const token = environment.ORCA_RELAY_ADMIN_ID_TOKEN
|
||||
if (!token || token.length > 8_192 || !/^[^.]+\.[^.]+\.[^.]+$/.test(token)) {
|
||||
throw new Error('admin identity token is unavailable')
|
||||
}
|
||||
return {
|
||||
directorOrigin: DIRECTOR_ORIGIN,
|
||||
cellId: values['cell-id'],
|
||||
cellIncarnation: values['cell-incarnation'],
|
||||
token
|
||||
}
|
||||
}
|
||||
|
||||
export async function probeRehomeTrust(config, dependencies = {}) {
|
||||
const fetchImpl = dependencies.fetch ?? fetch
|
||||
const request = () => fetchAdminOnceMore(
|
||||
fetchImpl,
|
||||
`${config.directorOrigin}/v1/admin/regional-rehome-trust-probe`,
|
||||
{
|
||||
method: 'POST',
|
||||
headers: {
|
||||
authorization: `Bearer ${config.token}`,
|
||||
'content-type': 'application/json'
|
||||
},
|
||||
body: JSON.stringify({
|
||||
v: 1,
|
||||
sourceCellId: config.cellId,
|
||||
sourceCellIncarnation: config.cellIncarnation
|
||||
})
|
||||
},
|
||||
{ wait: dependencies.wait }
|
||||
)
|
||||
const wait = dependencies.wait ?? ((ms) => new Promise((resolve) => setTimeout(resolve, ms)))
|
||||
let response = await request()
|
||||
let body = await response.json().catch(() => ({}))
|
||||
// The director wraps source HTTP failures in 409; retry only explicit transient statuses. The
|
||||
// load balancer can answer 503 for a minute after a fresh instance is ready (c10, 2026-10-20).
|
||||
for (
|
||||
let retry = 0;
|
||||
retry < SOURCE_TRANSIENT_RETRIES &&
|
||||
response.status === 409 &&
|
||||
/^regional_rehome_trust_probe_source_(500|502|503|504)$/.test(body?.error ?? '');
|
||||
retry++
|
||||
) {
|
||||
await wait(SOURCE_TRANSIENT_RETRY_DELAY_MS)
|
||||
response = await request()
|
||||
body = await response.json().catch(() => ({}))
|
||||
}
|
||||
if (!response.ok) {
|
||||
const safeReasons = new Set([
|
||||
'invalid_token', 'director_only', 'invalid_request',
|
||||
'regional_rehome_trust_not_configured',
|
||||
'regional_rehome_trust_probe_source_unavailable',
|
||||
'regional_rehome_trust_probe_source_invalid_response',
|
||||
'regional_rehome_trust_probe_not_proven',
|
||||
...[400, 401, 403, 404, 409, 429, 500, 502, 503, 504]
|
||||
.map((status) => `regional_rehome_trust_probe_source_${status}`)
|
||||
])
|
||||
const reason = safeReasons.has(body?.error) ? body.error : 'unrecognized_error'
|
||||
throw new Error(`application-mediated rehome trust probe returned ${response.status}: ${reason}`)
|
||||
}
|
||||
if (
|
||||
body.v !== 1 ||
|
||||
body.dedicatedIdentity?.firstOutcome !== 'host-not-connected' ||
|
||||
body.dedicatedIdentity?.secondOutcome !== 'host-not-connected' ||
|
||||
body.dedicatedIdentity?.accepted !== true ||
|
||||
body.dedicatedIdentity?.idempotent !== true ||
|
||||
body.sharedRuntimeIdentityRejected !== true ||
|
||||
body.proven !== true
|
||||
) throw new Error('application-mediated rehome trust proof is incomplete')
|
||||
return body
|
||||
}
|
||||
|
||||
export async function main(argv = process.argv.slice(2)) {
|
||||
const result = await probeRehomeTrust(parseRehomeTrustProbeArguments(argv))
|
||||
process.stdout.write(`${JSON.stringify({ event: 'relay_rehome_trust_verified', ...result })}\n`)
|
||||
}
|
||||
|
||||
if (import.meta.url === pathToFileURL(process.argv[1]).href) {
|
||||
main().catch((error) => {
|
||||
process.stderr.write(`${error instanceof Error ? error.message : String(error)}\n`)
|
||||
process.exitCode = 1
|
||||
})
|
||||
}
|
||||
@@ -1,217 +0,0 @@
|
||||
import assert from 'node:assert/strict'
|
||||
import { readFileSync } from 'node:fs'
|
||||
import { test } from 'node:test'
|
||||
import {
|
||||
parseRehomeTrustProbeArguments,
|
||||
probeRehomeTrust
|
||||
} from './probe-relay-rehome-trust.mjs'
|
||||
|
||||
const argv = [
|
||||
'--director-origin', 'https://relay.onorca.dev',
|
||||
'--cell-id', 'production-gce-c7',
|
||||
'--cell-incarnation', '11111111-1111-4111-8111-111111111111'
|
||||
]
|
||||
const environment = { ORCA_RELAY_ADMIN_ID_TOKEN: 'aaa.bbb.ccc' }
|
||||
|
||||
test('binds the application-mediated probe to an exact approved cell incarnation', () => {
|
||||
assert.equal(parseRehomeTrustProbeArguments(argv, environment).cellId, 'production-gce-c7')
|
||||
assert.throws(() => parseRehomeTrustProbeArguments(
|
||||
argv.with(1, 'https://other.example.test'),
|
||||
environment
|
||||
))
|
||||
assert.throws(() => parseRehomeTrustProbeArguments(argv, {
|
||||
ORCA_RELAY_ADMIN_ID_TOKEN: 'not-a-token'
|
||||
}))
|
||||
})
|
||||
|
||||
test('requires complete aggregate application-mediated trust proof', async () => {
|
||||
const config = parseRehomeTrustProbeArguments(argv, environment)
|
||||
const result = await probeRehomeTrust(config, {
|
||||
fetch: async (url, init) => {
|
||||
assert.equal(url, 'https://relay.onorca.dev/v1/admin/regional-rehome-trust-probe')
|
||||
assert.deepEqual(JSON.parse(init.body), {
|
||||
v: 1,
|
||||
sourceCellId: 'production-gce-c7',
|
||||
sourceCellIncarnation: '11111111-1111-4111-8111-111111111111'
|
||||
})
|
||||
return Response.json({
|
||||
v: 1,
|
||||
dedicatedIdentity: {
|
||||
firstOutcome: 'host-not-connected',
|
||||
secondOutcome: 'host-not-connected',
|
||||
accepted: true,
|
||||
idempotent: true
|
||||
},
|
||||
sharedRuntimeIdentityRejected: true,
|
||||
proven: true
|
||||
})
|
||||
}
|
||||
})
|
||||
assert.equal(result.proven, true)
|
||||
})
|
||||
|
||||
test('rejects partial or mismatched proof', async () => {
|
||||
const config = parseRehomeTrustProbeArguments(argv, environment)
|
||||
await assert.rejects(
|
||||
probeRehomeTrust(config, {
|
||||
fetch: async () => Response.json({
|
||||
v: 1,
|
||||
dedicatedIdentity: {
|
||||
firstOutcome: 'host-not-connected',
|
||||
secondOutcome: 'host-not-connected',
|
||||
accepted: true,
|
||||
idempotent: true
|
||||
},
|
||||
sharedRuntimeIdentityRejected: false,
|
||||
proven: false
|
||||
})
|
||||
}),
|
||||
/incomplete/
|
||||
)
|
||||
})
|
||||
|
||||
const provenProbe = {
|
||||
v: 1,
|
||||
dedicatedIdentity: {
|
||||
firstOutcome: 'host-not-connected',
|
||||
secondOutcome: 'host-not-connected',
|
||||
accepted: true,
|
||||
idempotent: true
|
||||
},
|
||||
sharedRuntimeIdentityRejected: true,
|
||||
proven: true
|
||||
}
|
||||
|
||||
test('retries a transient 503 on the trust probe and proves on the second answer', async () => {
|
||||
const config = parseRehomeTrustProbeArguments(argv, environment)
|
||||
let calls = 0
|
||||
const result = await probeRehomeTrust(config, {
|
||||
wait: async () => {},
|
||||
fetch: async () => {
|
||||
calls += 1
|
||||
if (calls === 1) return new Response('warming up', { status: 503 })
|
||||
return Response.json(provenProbe)
|
||||
}
|
||||
})
|
||||
assert.equal(calls, 2)
|
||||
assert.equal(result.proven, true)
|
||||
})
|
||||
|
||||
test('fails when both trust-probe attempts return a transient 503', async () => {
|
||||
const config = parseRehomeTrustProbeArguments(argv, environment)
|
||||
let calls = 0
|
||||
await assert.rejects(
|
||||
probeRehomeTrust(config, {
|
||||
wait: async () => {},
|
||||
fetch: async () => {
|
||||
calls += 1
|
||||
return new Response('warming up', { status: 503 })
|
||||
}
|
||||
}),
|
||||
/returned 503/
|
||||
)
|
||||
assert.equal(calls, 2)
|
||||
})
|
||||
|
||||
test('approves the asia-east2 and US 3,000 rehome sources and still rejects unlisted cells', () => {
|
||||
for (const cellId of [
|
||||
'production-gce-c27', 'production-gce-c28', 'production-gce-c29', 'production-gce-c30',
|
||||
'production-gce-c31', 'production-gce-c32', 'production-gce-c33', 'production-gce-c34',
|
||||
'production-gce-c35'
|
||||
]) {
|
||||
const parsed = parseRehomeTrustProbeArguments(
|
||||
argv.map((value) => (value === 'production-gce-c7' ? cellId : value)),
|
||||
environment
|
||||
)
|
||||
assert.equal(parsed.cellId, cellId)
|
||||
}
|
||||
for (const cellId of ['production-gce-c1', 'production-gce-c17', 'production-gce-c36']) {
|
||||
assert.throws(
|
||||
() =>
|
||||
parseRehomeTrustProbeArguments(
|
||||
argv.map((value) => (value === 'production-gce-c7' ? cellId : value)),
|
||||
environment
|
||||
),
|
||||
/--cell-id is not approved/
|
||||
)
|
||||
}
|
||||
})
|
||||
|
||||
test('retries one director-wrapped source 503 without relaxing the proof', async () => {
|
||||
let calls = 0
|
||||
const result = await probeRehomeTrust(parseRehomeTrustProbeArguments(argv, environment), {
|
||||
wait: async () => {},
|
||||
fetch: async () => ++calls === 1
|
||||
? Response.json({ error: 'regional_rehome_trust_probe_source_503' }, { status: 409 })
|
||||
: Response.json(provenProbe)
|
||||
})
|
||||
assert.equal(calls, 2)
|
||||
assert.equal(result.proven, true)
|
||||
})
|
||||
|
||||
test('reports safe trust reasons, keeps rejection final, and redacts arbitrary error text', async () => {
|
||||
for (const reason of ['regional_rehome_trust_probe_source_403', 'secret-token-example']) {
|
||||
let calls = 0
|
||||
await assert.rejects(probeRehomeTrust(parseRehomeTrustProbeArguments(argv, environment), {
|
||||
wait: async () => { throw new Error('must not retry') },
|
||||
fetch: async () => { calls++; return Response.json({ error: reason }, { status: 409 }) }
|
||||
}), error => {
|
||||
assert.match(error.message, /returned 409/)
|
||||
assert.ok(!error.message.includes('secret-token-example'))
|
||||
if (reason.endsWith('_403')) assert.match(error.message, /source_403/)
|
||||
return true
|
||||
})
|
||||
assert.equal(calls, 1)
|
||||
}
|
||||
})
|
||||
|
||||
test('outlasts a load-balancer warm-up of consecutive wrapped source 503s', async () => {
|
||||
let calls = 0
|
||||
const waits = []
|
||||
const result = await probeRehomeTrust(parseRehomeTrustProbeArguments(argv, environment), {
|
||||
wait: async (ms) => { waits.push(ms) },
|
||||
fetch: async () => ++calls <= 6
|
||||
? Response.json({ error: 'regional_rehome_trust_probe_source_503' }, { status: 409 })
|
||||
: Response.json(provenProbe)
|
||||
})
|
||||
assert.equal(calls, 7)
|
||||
assert.deepEqual(waits, Array(6).fill(10_000))
|
||||
assert.equal(result.proven, true)
|
||||
})
|
||||
|
||||
test('stops after two minutes of wrapped transient failures', async () => {
|
||||
let calls = 0
|
||||
await assert.rejects(probeRehomeTrust(parseRehomeTrustProbeArguments(argv, environment), {
|
||||
wait: async () => {},
|
||||
fetch: async () => { calls++; return Response.json({ error: 'regional_rehome_trust_probe_source_503' }, { status: 409 }) }
|
||||
}), /returned 409.*source_503/)
|
||||
assert.equal(calls, 13)
|
||||
})
|
||||
|
||||
// The probe allowlist is the committed rehome source list, so a declared source can always probe.
|
||||
test('approves exactly the committed production rehome source cells', () => {
|
||||
const tfvars = readFileSync(
|
||||
new URL('../../infra/terraform/environments/production.tfvars', import.meta.url),
|
||||
'utf8'
|
||||
)
|
||||
const start = tfvars.indexOf('relay_region_rehome_source_cell_ids = [')
|
||||
const sources = new Set(
|
||||
[...tfvars.slice(start, tfvars.indexOf(']', start)).matchAll(/"([^"]+)"/g)].map(([, cell]) => cell)
|
||||
)
|
||||
assert.ok(sources.has('production-gce-c35'))
|
||||
for (let ordinal = 1; ordinal <= 40; ordinal++) {
|
||||
const cellId = `production-gce-c${ordinal}`
|
||||
const approved = (() => {
|
||||
try {
|
||||
parseRehomeTrustProbeArguments(
|
||||
argv.map((value) => (value === 'production-gce-c7' ? cellId : value)),
|
||||
environment
|
||||
)
|
||||
return true
|
||||
} catch {
|
||||
return false
|
||||
}
|
||||
})()
|
||||
assert.equal(approved, sources.has(cellId), cellId)
|
||||
}
|
||||
})
|
||||
@@ -1,85 +0,0 @@
|
||||
import assert from 'node:assert/strict'
|
||||
import { readFileSync } from 'node:fs'
|
||||
import test from 'node:test'
|
||||
import { PRODUCTION_CAPACITY_CELL_IDS } from './prepare-relay-production-capacity-canary.mjs'
|
||||
import { readRelayWorkflow } from './relay-repository.mjs'
|
||||
|
||||
const production = source('infra/terraform/environments/production.tfvars')
|
||||
const dispatchWorkflow = readRelayWorkflow('deploy-relay-production-capacity.yml')
|
||||
const jobWorkflow = readRelayWorkflow('deploy-relay-production-capacity-job.yml')
|
||||
|
||||
const RELAY_REPOSITORY = 'us-central1-docker.pkg.dev/onorca-cloud/orca-cloud/relay'
|
||||
|
||||
function source(path) {
|
||||
return readFileSync(new URL(`../../${path}`, import.meta.url), 'utf8')
|
||||
}
|
||||
|
||||
// Slice each "<cell-id>" = { ... } entry out of relay_gce_cells.
|
||||
function productionCells() {
|
||||
const block = production.slice(production.indexOf('relay_gce_cells = {'))
|
||||
const cells = new Map()
|
||||
for (const match of block.matchAll(/"(production-gce-c\d+)" = \{([\s\S]*?)\n {2}\}/g)) {
|
||||
cells.set(match[1], match[2])
|
||||
}
|
||||
assert.ok(cells.size > 0, 'relay_gce_cells parsed empty')
|
||||
return cells
|
||||
}
|
||||
|
||||
function hardCap(body) {
|
||||
const match = body.match(/connection_hard_cap\s*=\s*(\d+)/)
|
||||
return match ? Number(match[1]) : undefined
|
||||
}
|
||||
|
||||
function imageDigest(body) {
|
||||
const match = body.match(/^\s*image\s*=\s*"([^"]+)"/m)
|
||||
assert.ok(match, 'cell entry has no image')
|
||||
const [repository, digest] = match[1].split('@')
|
||||
assert.equal(repository, RELAY_REPOSITORY)
|
||||
assert.match(digest, /^sha256:[0-9a-f]{64}$/)
|
||||
return digest
|
||||
}
|
||||
|
||||
function workflowPin(workflow, name) {
|
||||
const match = workflow.match(new RegExp(`${name}: (sha256:[0-9a-f]{64})`))
|
||||
assert.ok(match, `${name} is missing or not a full digest`)
|
||||
return match[1]
|
||||
}
|
||||
|
||||
test('every production cell pins a full relay image digest', () => {
|
||||
for (const [cellId, body] of productionCells()) {
|
||||
assert.match(imageDigest(body), /^sha256:[0-9a-f]{64}$/, `${cellId} image digest`)
|
||||
}
|
||||
})
|
||||
|
||||
test('the 1,000-cap cells are exactly the canonical capacity set', () => {
|
||||
const thousandCap = [...productionCells()]
|
||||
.filter(([, body]) => hardCap(body) === 1000)
|
||||
.map(([cellId]) => cellId)
|
||||
assert.deepEqual([...thousandCap].sort(), [...PRODUCTION_CAPACITY_CELL_IDS].sort())
|
||||
})
|
||||
|
||||
test('the 1,000-cap cells all serve one image digest', () => {
|
||||
const digests = new Map()
|
||||
for (const [cellId, body] of productionCells()) {
|
||||
if (hardCap(body) !== 1000) continue
|
||||
const digest = imageDigest(body)
|
||||
if (!digests.has(digest)) digests.set(digest, [])
|
||||
digests.get(digest).push(cellId)
|
||||
}
|
||||
assert.equal(
|
||||
digests.size,
|
||||
1,
|
||||
`1,000-cap cells split across digests: ${JSON.stringify([...digests])}`
|
||||
)
|
||||
assert.equal([...digests.values()][0].length, PRODUCTION_CAPACITY_CELL_IDS.length)
|
||||
})
|
||||
|
||||
// COMPATIBLE_CELL_IMAGE_DIGEST is one half of a reviewed (director, cell) skew pair, not a
|
||||
// claim about what the fleet serves; it is re-derived by hand for each capacity wave. So it
|
||||
// is deliberately NOT tied to the tfvars digest — only to its twin in the dispatch workflow.
|
||||
test('both capacity workflows declare the same reviewed image pins', () => {
|
||||
for (const name of ['PREDECESSOR_IMAGE_DIGEST', 'COMPATIBLE_CELL_IMAGE_DIGEST']) {
|
||||
assert.equal(workflowPin(dispatchWorkflow, name), workflowPin(jobWorkflow, name), name)
|
||||
}
|
||||
workflowPin(jobWorkflow, 'COMPATIBLE_DIRECTOR_IMAGE_DIGEST')
|
||||
})
|
||||
@@ -1,182 +0,0 @@
|
||||
import assert from 'node:assert/strict'
|
||||
import { test } from 'node:test'
|
||||
import {
|
||||
LEASED_WORKFLOWS,
|
||||
LOCK_GROUPS,
|
||||
NOT_A_CLOUD_SQL_CANDIDATE,
|
||||
PRODUCTION_LEASE,
|
||||
SELECTABLE_LEASE,
|
||||
STAGING_LEASE,
|
||||
concurrencyBlocks,
|
||||
entrypointsFor,
|
||||
jobIf,
|
||||
jobNeeds,
|
||||
jobs,
|
||||
leaseSteps,
|
||||
leaseStepsByJob,
|
||||
mutatesSharedInstance,
|
||||
readWorkflow,
|
||||
reusableCalls,
|
||||
revisionMintingScripts,
|
||||
workflowFiles
|
||||
} from './cloud-sql-rollout-lock-census.mjs'
|
||||
|
||||
const expectedLease = { production: PRODUCTION_LEASE, staging: STAGING_LEASE, selectable: SELECTABLE_LEASE }
|
||||
const leasedFiles = Object.keys(LEASED_WORKFLOWS)
|
||||
|
||||
function contractFiles(file) {
|
||||
return [file, ...(LEASED_WORKFLOWS[file].leaseFiles ?? [])]
|
||||
}
|
||||
|
||||
test('every locked workflow declares exactly one lock group that never cancels', () => {
|
||||
for (const file of leasedFiles) {
|
||||
const blocks = concurrencyBlocks(readWorkflow(file))
|
||||
assert.equal(blocks.length, 1, `${file} must declare exactly one concurrency block`)
|
||||
assert.equal(blocks[0].group, LEASED_WORKFLOWS[file].group, file)
|
||||
assert.equal(blocks[0].cancelInProgress, 'false', file)
|
||||
}
|
||||
})
|
||||
|
||||
test('every locked workflow takes the lease for its own environment', () => {
|
||||
for (const file of leasedFiles) {
|
||||
const lease = expectedLease[LEASED_WORKFLOWS[file].env]
|
||||
const steps = contractFiles(file).flatMap((member) => leaseSteps(readWorkflow(member)))
|
||||
assert.ok(steps.length > 0, `${file} must use the Cloud SQL rollout lease action`)
|
||||
for (const step of steps) {
|
||||
assert.equal(step.bucket, lease.bucket, file)
|
||||
assert.equal(step.object, lease.object, file)
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
test('the lease step runs after the credential that authorizes it', () => {
|
||||
for (const file of leasedFiles) {
|
||||
for (const member of contractFiles(file)) {
|
||||
const text = readWorkflow(member)
|
||||
const lines = text.split('\n')
|
||||
for (const step of leaseSteps(text)) {
|
||||
const before = lines.slice(0, step.line - 1)
|
||||
const gcloud = before.lastIndexOf(' - uses: google-github-actions/setup-gcloud@v2')
|
||||
assert.notEqual(gcloud, -1, `${member}: lease step at line ${step.line} has no setup-gcloud before it`)
|
||||
assert.ok(
|
||||
before.lastIndexOf(' - uses: actions/checkout@v4') !== -1,
|
||||
`${member}: lease step at line ${step.line} runs before the local action is checked out`
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
test('multi-wave workflows hold one lease per run and free it exactly once', () => {
|
||||
for (const file of leasedFiles) {
|
||||
const entry = LEASED_WORKFLOWS[file]
|
||||
const waveFiles = entry.leaseFiles ?? []
|
||||
const callCount = waveFiles.reduce(
|
||||
(total, member) => total + (reusableCalls(readWorkflow(file)).get(member) ?? 0),
|
||||
0
|
||||
)
|
||||
if (!entry.reentrant) {
|
||||
assert.ok(callCount <= 1, `${file} calls a leased reusable job ${callCount} times; it needs a release job`)
|
||||
for (const member of contractFiles(file)) {
|
||||
for (const step of leaseSteps(readWorkflow(member))) {
|
||||
assert.equal(step.release, undefined, `${member} must leave release at its default`)
|
||||
}
|
||||
}
|
||||
continue
|
||||
}
|
||||
|
||||
assert.ok(callCount > 1, `${file} no longer calls its reusable job more than once`)
|
||||
const steps = contractFiles(file).flatMap((member) => leaseSteps(readWorkflow(member)))
|
||||
const released = steps.filter((step) => step.release === "'true'")
|
||||
assert.equal(released.length, 1, `${file} must free the run lease exactly once`)
|
||||
for (const step of steps) {
|
||||
if (step === released[0]) continue
|
||||
assert.equal(step.release, "'false'", `${file} wave jobs must hold the lease`)
|
||||
}
|
||||
|
||||
const callerJobs = jobs(readWorkflow(file))
|
||||
const releaseJob = leaseStepsByJob(file).find((job) =>
|
||||
job.steps.some((step) => step.release === "'true'")
|
||||
)
|
||||
assert.ok(releaseJob, `${file} must free the lease from its own job`)
|
||||
const guard = jobIf(callerJobs.find((job) => job.id === releaseJob.id).text)
|
||||
assert.match(guard, /always\(\)/, `${file}: the release job must run on failure and cancellation`)
|
||||
|
||||
const holders = callerJobs
|
||||
.filter(
|
||||
(job) =>
|
||||
job.id !== releaseJob.id &&
|
||||
(waveFiles.some((member) => job.text.includes(`uses: ./.github/workflows/${member}`)) ||
|
||||
leaseStepsByJob(file).find((entry) => entry.id === job.id)?.steps.length > 0)
|
||||
)
|
||||
.map((job) => job.id)
|
||||
const needs = jobNeeds(callerJobs.find((job) => job.id === releaseJob.id).text)
|
||||
for (const holder of holders) {
|
||||
assert.ok(needs.includes(holder), `${file}: the release job must need ${holder}`)
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
test('workflows with two lease-holding jobs can never run them together', () => {
|
||||
for (const file of leasedFiles) {
|
||||
const entry = LEASED_WORKFLOWS[file]
|
||||
if (entry.reentrant) continue
|
||||
const holding = leaseStepsByJob(file).filter((job) => job.steps.length > 0)
|
||||
if (holding.length <= 1) continue
|
||||
assert.ok(entry.exclusiveBy, `${file} has ${holding.length} lease-holding jobs and no exclusivity guard`)
|
||||
const guards = holding.map((job) => jobIf(jobs(readWorkflow(file)).find((j) => j.id === job.id).text))
|
||||
assert.equal(
|
||||
guards.filter((guard) => guard.includes(entry.exclusiveBy)).length,
|
||||
1,
|
||||
`${file}: exactly one job may run when ${entry.exclusiveBy}`
|
||||
)
|
||||
assert.equal(
|
||||
guards.filter((guard) => guard.includes(entry.exclusiveBy.replace('==', '!='))).length,
|
||||
guards.length - 1,
|
||||
`${file}: every other lease-holding job must be excluded when ${entry.exclusiveBy}`
|
||||
)
|
||||
}
|
||||
})
|
||||
|
||||
test('census: no workflow rolls out against the shared instance outside the lease', () => {
|
||||
const minters = revisionMintingScripts()
|
||||
assert.ok(minters.size > 0, 'the revision-minting script scan found nothing and is vacuous')
|
||||
const flagged = new Map()
|
||||
for (const file of workflowFiles()) {
|
||||
const reason = mutatesSharedInstance(readWorkflow(file), minters)
|
||||
if (reason) flagged.set(file, reason)
|
||||
}
|
||||
assert.ok(flagged.size > 0, 'the rollout census found no candidates and is vacuous')
|
||||
|
||||
for (const [file, reason] of flagged) {
|
||||
for (const entrypoint of entrypointsFor(file)) {
|
||||
assert.ok(
|
||||
entrypoint in LEASED_WORKFLOWS || entrypoint in NOT_A_CLOUD_SQL_CANDIDATE,
|
||||
`${entrypoint} reaches ${file} (${reason}) but is neither leased nor recorded as a non-candidate`
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
for (const file of workflowFiles()) {
|
||||
const groups = concurrencyBlocks(readWorkflow(file)).map((block) => block.group)
|
||||
if (!groups.some((group) => LOCK_GROUPS.has(group))) continue
|
||||
assert.ok(
|
||||
file in LEASED_WORKFLOWS || file in NOT_A_CLOUD_SQL_CANDIDATE,
|
||||
`${file} sits in a Cloud SQL lock group but is neither leased nor recorded as a non-candidate`
|
||||
)
|
||||
}
|
||||
|
||||
for (const [file, reason] of Object.entries(NOT_A_CLOUD_SQL_CANDIDATE)) {
|
||||
assert.ok(typeof reason === 'string' && reason.length > 40, `${file} needs a real reason`)
|
||||
const groups = concurrencyBlocks(readWorkflow(file)).map((block) => block.group)
|
||||
assert.ok(
|
||||
flagged.has(file) || groups.some((group) => LOCK_GROUPS.has(group)),
|
||||
`${file} is recorded as a non-candidate but nothing would have flagged it`
|
||||
)
|
||||
}
|
||||
|
||||
for (const file of leasedFiles) {
|
||||
assert.doesNotThrow(() => readWorkflow(file), `${file} is leased but does not exist`)
|
||||
assert.ok(!(file in NOT_A_CLOUD_SQL_CANDIDATE), `${file} cannot be both leased and a non-candidate`)
|
||||
}
|
||||
})
|
||||
@@ -1,140 +0,0 @@
|
||||
import { readFileSync, writeFileSync } from 'node:fs'
|
||||
|
||||
// Executable fake gcloud: revision deletion obeys the platform's latest/traffic constraints.
|
||||
const path = process.env.MODEL_STATE
|
||||
const state = JSON.parse(readFileSync(path, 'utf8'))
|
||||
const args = process.argv.slice(2)
|
||||
const option = (name) => args[args.indexOf(name) + 1]
|
||||
const has = (name) => args.includes(name)
|
||||
const fail = (message) => {
|
||||
throw new Error(message)
|
||||
}
|
||||
const persist = () => writeFileSync(path, JSON.stringify(state))
|
||||
const output = (value) => console.log(typeof value === 'string' ? value : JSON.stringify(value))
|
||||
const revision = (name) => state.revisions[name] ?? fail(`missing revision ${name}`)
|
||||
const traffic = () => [
|
||||
{ revisionName: state.serving, percent: 100 },
|
||||
...Object.entries(state.tags).map(([tag, name]) => ({
|
||||
tag,
|
||||
revisionName: name,
|
||||
url: `https://${tag}.test`
|
||||
}))
|
||||
]
|
||||
state.trace.push(args.join(' '))
|
||||
try {
|
||||
if (args[0] === 'curl') {
|
||||
if (state.failure === 'public' && args.some((arg) => arg.includes('https://public.test'))) {
|
||||
fail('public check failed')
|
||||
}
|
||||
const url = args.find((arg) => arg.startsWith('https://'))
|
||||
const tag = new URL(url).hostname.split('.')[0]
|
||||
const name = state.tags[tag] ?? state.serving
|
||||
if (has('-w')) {
|
||||
output('200')
|
||||
} else {
|
||||
output({
|
||||
ok: true,
|
||||
deliveryProtocol: 2,
|
||||
mode: revision(name).spec.containers[0].env.some((entry) => entry.value === 'validation')
|
||||
? 'validation'
|
||||
: 'active'
|
||||
})
|
||||
}
|
||||
} else if (args.slice(0, 2).join(' ') === 'run deploy') {
|
||||
const name = `${option('deploy')}-${option('--revision-suffix')}`
|
||||
if (state.failure === 'deploy-before') {
|
||||
fail('deploy failed before create')
|
||||
}
|
||||
const item = structuredClone(revision(state.latest))
|
||||
item.metadata.name = name
|
||||
item.spec.containers[0].image = option('--image')
|
||||
item.status.imageDigest = option('--image')
|
||||
item.spec.containers[0].env = item.spec.containers[0].env.filter(
|
||||
(entry) => entry.name !== 'ORCA_PUSH_MODE'
|
||||
)
|
||||
if (has('--update-env-vars')) {
|
||||
item.spec.containers[0].env.push({ name: 'ORCA_PUSH_MODE', value: 'validation' })
|
||||
}
|
||||
state.revisions[name] = item
|
||||
state.latest = name
|
||||
if (has('--tag')) {
|
||||
state.tags[option('--tag')] = name
|
||||
}
|
||||
state.peak = Math.max(state.peak, Object.keys(state.revisions).length)
|
||||
if (state.peak > 3) {
|
||||
fail('three-revision budget exceeded')
|
||||
}
|
||||
if (state.failure === 'deploy-after') {
|
||||
fail('deploy failed after create')
|
||||
}
|
||||
} else if (args.slice(0, 3).join(' ') === 'run services describe') {
|
||||
if (state.failure === 'describe') {
|
||||
fail('describe failed')
|
||||
}
|
||||
if (args.some((arg) => arg.includes('value(status.latestCreatedRevisionName)'))) {
|
||||
output(state.latest)
|
||||
} else {
|
||||
const template = structuredClone(revision(state.latest))
|
||||
delete template.spec.containers[0].name
|
||||
output({
|
||||
spec: { template },
|
||||
status: { latestCreatedRevisionName: state.latest, traffic: traffic() }
|
||||
})
|
||||
}
|
||||
} else if (args.slice(0, 3).join(' ') === 'run services update-traffic') {
|
||||
if (has('--to-revisions')) {
|
||||
const name = option('--to-revisions').split('=')[0]
|
||||
revision(name)
|
||||
state.serving = name
|
||||
}
|
||||
if (has('--remove-tags')) {
|
||||
for (const tag of option('--remove-tags').split(',')) {
|
||||
delete state.tags[tag]
|
||||
}
|
||||
}
|
||||
if (has('--clear-tags')) {
|
||||
state.tags = {}
|
||||
}
|
||||
if (state.failure === 'traffic-after') {
|
||||
fail('traffic changed but response failed')
|
||||
}
|
||||
} else if (args.slice(0, 3).join(' ') === 'run revisions list') {
|
||||
const names = Object.keys(state.revisions)
|
||||
output(
|
||||
(has('--filter')
|
||||
? names.filter((name) => name === option('--filter').split('=')[1])
|
||||
: names
|
||||
).join('\n')
|
||||
)
|
||||
} else if (args.slice(0, 3).join(' ') === 'run revisions describe') {
|
||||
const item = revision(args[3])
|
||||
const format = args.find((arg) => arg.startsWith('--format=')) ?? option('--format')
|
||||
if (format.includes('minScale')) {
|
||||
output(item.metadata.annotations['autoscaling.knative.dev/minScale'])
|
||||
} else if (format.includes('maxScale')) {
|
||||
output(item.metadata.annotations['autoscaling.knative.dev/maxScale'])
|
||||
} else {
|
||||
output(item)
|
||||
}
|
||||
} else if (args.slice(0, 3).join(' ') === 'run revisions delete') {
|
||||
const name = args[3]
|
||||
if (name === state.latest) {
|
||||
fail('FAILED_PRECONDITION: latest created Revision cannot be directly deleted')
|
||||
}
|
||||
if (name === state.serving || Object.values(state.tags).includes(name)) {
|
||||
fail('revision has traffic or tags')
|
||||
}
|
||||
if (state.failure === 'delete') {
|
||||
fail('delete failed')
|
||||
}
|
||||
revision(name)
|
||||
delete state.revisions[name]
|
||||
} else {
|
||||
fail(`unmodeled gcloud call ${args.join(' ')}`)
|
||||
}
|
||||
} catch (error) {
|
||||
console.error(error.message)
|
||||
process.exitCode = 1
|
||||
} finally {
|
||||
persist()
|
||||
}
|
||||
@@ -1,373 +0,0 @@
|
||||
import assert from 'node:assert/strict'
|
||||
import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { join } from 'node:path'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
import { spawnSync } from 'node:child_process'
|
||||
import test from 'node:test'
|
||||
import { readRelayWorkflow } from './relay-repository.mjs'
|
||||
|
||||
const workflow = readRelayWorkflow('push-deploy.yml')
|
||||
function step(name) {
|
||||
const start = workflow.indexOf(` - name: ${name}\n`)
|
||||
assert.notEqual(start, -1)
|
||||
const end = workflow.indexOf('\n - ', start + 1)
|
||||
const block = workflow.slice(start, end === -1 ? undefined : end)
|
||||
return block
|
||||
.slice(block.indexOf(' run: |\n') + ' run: |\n'.length)
|
||||
.split('\n')
|
||||
.filter((line) => line.startsWith(' '))
|
||||
.map((line) => line.slice(10))
|
||||
.join('\n')
|
||||
}
|
||||
const names = {
|
||||
preflight: 'Record the serving revision and require its Terraform-owned scaling',
|
||||
candidate: 'Deploy the candidate revision with no traffic',
|
||||
activate: 'Retire inert validation and activate the verified image',
|
||||
shift: 'Shift all traffic to the verified candidate',
|
||||
public: 'Verify the public origin after the shift',
|
||||
rollback: 'Roll traffic back to the previous revision',
|
||||
restore: 'Restore the known-good service template',
|
||||
promoteRecovery: 'Promote and verify the known-good recovery revision',
|
||||
cleanup: 'Delete the rejected candidate revision',
|
||||
retire: 'Retire previous consumers after public checks'
|
||||
}
|
||||
const image = `registry/push@sha256:${'a'.repeat(64)}`
|
||||
const spec = {
|
||||
serviceAccountName: 'runtime@test',
|
||||
containerConcurrency: 40,
|
||||
containers: [
|
||||
{
|
||||
image,
|
||||
name: 'push-test-1',
|
||||
env: [
|
||||
{
|
||||
name: 'ORCA_PUSH_DATABASE_URL',
|
||||
valueFrom: { secretKeyRef: { name: 'database', key: '7' } }
|
||||
},
|
||||
{ name: 'ORCA_PUSH_DATABASE_POOL_MAX', value: '2' }
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
const prior = {
|
||||
metadata: {
|
||||
name: 'push-test-old',
|
||||
annotations: {
|
||||
'autoscaling.knative.dev/minScale': '1',
|
||||
'autoscaling.knative.dev/maxScale': '2'
|
||||
}
|
||||
},
|
||||
spec,
|
||||
status: { imageDigest: image }
|
||||
}
|
||||
const model = fileURLToPath(new URL('./push-cloud-run-model.mjs', import.meta.url))
|
||||
const options = { skip: process.platform === 'win32' }
|
||||
function exercise(callback) {
|
||||
const dir = mkdtempSync(join(tmpdir(), 'push-workflow-'))
|
||||
const statePath = join(dir, 'state.json')
|
||||
writeFileSync(
|
||||
statePath,
|
||||
JSON.stringify({
|
||||
revisions: { 'push-test-old': prior },
|
||||
latest: 'push-test-old',
|
||||
serving: 'push-test-old',
|
||||
tags: {},
|
||||
peak: 1,
|
||||
trace: []
|
||||
})
|
||||
)
|
||||
writeFileSync(join(dir, 'env'), '')
|
||||
const env = {
|
||||
...process.env,
|
||||
SERVICE_NAME: 'push-test',
|
||||
GCP_PROJECT_ID: 'test',
|
||||
GCP_REGION: 'test',
|
||||
GITHUB_RUN_ID: '123',
|
||||
GITHUB_RUN_ATTEMPT: '1',
|
||||
IMAGE: `registry/push@sha256:${'b'.repeat(64)}`,
|
||||
PUSH_MIN_INSTANCES: '1',
|
||||
PUSH_MAX_INSTANCES: '2',
|
||||
PUSH_RUNTIME_SERVICE_ACCOUNT: 'runtime@test',
|
||||
PUSH_ORIGIN: 'https://public.test',
|
||||
MODEL_STATE: statePath,
|
||||
MODEL_SCRIPT: model,
|
||||
RUNNER_TEMP: dir,
|
||||
GITHUB_ENV: join(dir, 'env'),
|
||||
GITHUB_STEP_SUMMARY: join(dir, 'summary')
|
||||
}
|
||||
const state = () => JSON.parse(readFileSync(statePath, 'utf8'))
|
||||
const change = (edit) => {
|
||||
const value = state()
|
||||
edit(value)
|
||||
writeFileSync(statePath, JSON.stringify(value))
|
||||
}
|
||||
const run = (key, ok = true, extra = '') => {
|
||||
const result = spawnSync(
|
||||
'bash',
|
||||
[
|
||||
'-c',
|
||||
`
|
||||
set -a
|
||||
source "$GITHUB_ENV"
|
||||
gcloud() { node "$MODEL_SCRIPT" "$@"; }
|
||||
curl() { node "$MODEL_SCRIPT" curl "$@"; }
|
||||
sleep() { :; }
|
||||
${extra}
|
||||
${names[key] ? step(names[key]) : key}
|
||||
`
|
||||
],
|
||||
{ cwd: dir, env, encoding: 'utf8', timeout: 30000 }
|
||||
)
|
||||
assert.equal(result.status === 0, ok, `${key}: ${result.stderr}\n${result.stdout}`)
|
||||
return result
|
||||
}
|
||||
try {
|
||||
callback({ run, state, change, dir })
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true })
|
||||
}
|
||||
}
|
||||
function recover(h) {
|
||||
h.change((state) => {
|
||||
delete state.failure
|
||||
})
|
||||
h.run('restore')
|
||||
h.run('promoteRecovery')
|
||||
h.run('cleanup')
|
||||
h.run('retire')
|
||||
const state = h.state()
|
||||
assert.equal(state.serving, 'push-test-r123-1')
|
||||
assert.equal(state.latest, state.serving)
|
||||
assert.deepEqual(Object.keys(state.revisions), [state.serving])
|
||||
assert.equal(state.revisions[state.serving].spec.containers[0].image, image)
|
||||
assert.ok(state.peak <= 3)
|
||||
}
|
||||
|
||||
test(
|
||||
'the executable Cloud Run model rejects deleting latest even without tags or traffic',
|
||||
options,
|
||||
() =>
|
||||
exercise((h) => {
|
||||
h.run('preflight')
|
||||
h.run('candidate')
|
||||
h.run('gcloud run services update-traffic "$SERVICE_NAME" --clear-tags')
|
||||
const result = h.run('gcloud run revisions delete "$CANDIDATE_REVISION"', false)
|
||||
assert.match(result.stderr, /FAILED_PRECONDITION: latest created Revision/)
|
||||
})
|
||||
)
|
||||
|
||||
test(
|
||||
'success creates successor before retirement and repeated rollouts retain one consumer',
|
||||
options,
|
||||
() =>
|
||||
exercise((h) => {
|
||||
for (const attempt of ['1', '2']) {
|
||||
if (attempt === '2') {
|
||||
writeFileSync(join(h.dir, 'env'), 'GITHUB_RUN_ATTEMPT=2\n')
|
||||
}
|
||||
for (const key of ['preflight', 'candidate', 'activate', 'shift', 'public', 'retire']) {
|
||||
h.run(key)
|
||||
}
|
||||
const state = h.state()
|
||||
assert.equal(state.serving, `push-test-a123-${attempt}`)
|
||||
assert.deepEqual(Object.keys(state.revisions), [state.serving])
|
||||
assert.equal(state.peak, 3)
|
||||
}
|
||||
})
|
||||
)
|
||||
|
||||
for (const failure of ['deploy-before', 'deploy-after', 'describe']) {
|
||||
test(`validation ${failure} recovers without deleting latest`, options, () =>
|
||||
exercise((h) => {
|
||||
h.run('preflight')
|
||||
h.change((state) => {
|
||||
state.failure = failure
|
||||
})
|
||||
h.run('candidate', false)
|
||||
recover(h)
|
||||
})
|
||||
)
|
||||
}
|
||||
for (const failure of ['deploy-before', 'deploy-after', 'delete', 'describe']) {
|
||||
test(
|
||||
`activation ${failure} frees validation slot before recovery and stays within three`,
|
||||
options,
|
||||
() =>
|
||||
exercise((h) => {
|
||||
h.run('preflight')
|
||||
h.run('candidate')
|
||||
h.change((state) => {
|
||||
state.failure = failure
|
||||
})
|
||||
h.run('activate', false)
|
||||
recover(h)
|
||||
})
|
||||
)
|
||||
}
|
||||
|
||||
test(
|
||||
'ambiguous traffic shift records intent before mutation, rolls back and recovers',
|
||||
options,
|
||||
() =>
|
||||
exercise((h) => {
|
||||
h.run('preflight')
|
||||
h.run('candidate')
|
||||
h.run('activate')
|
||||
h.change((state) => {
|
||||
state.failure = 'traffic-after'
|
||||
})
|
||||
h.run('shift', false)
|
||||
assert.match(readFileSync(join(h.dir, 'env'), 'utf8'), /TRAFFIC_SHIFT_ATTEMPTED=true/)
|
||||
h.change((state) => {
|
||||
delete state.failure
|
||||
})
|
||||
h.run('rollback')
|
||||
recover(h)
|
||||
})
|
||||
)
|
||||
|
||||
test('failed public check rolls back and recovers', options, () =>
|
||||
exercise((h) => {
|
||||
for (const key of ['preflight', 'candidate', 'activate', 'shift']) {
|
||||
h.run(key)
|
||||
}
|
||||
h.change((state) => {
|
||||
state.failure = 'public'
|
||||
})
|
||||
h.run('public', false)
|
||||
h.change((state) => {
|
||||
delete state.failure
|
||||
})
|
||||
h.run('rollback')
|
||||
recover(h)
|
||||
})
|
||||
)
|
||||
|
||||
for (const defect of [
|
||||
'runtime',
|
||||
'secret',
|
||||
'mode',
|
||||
'image',
|
||||
'traffic',
|
||||
'scaling',
|
||||
'deploy-before',
|
||||
'deploy-after'
|
||||
]) {
|
||||
test(`recovery rejects ${defect} and preserves partial-create state`, options, () =>
|
||||
exercise((h) => {
|
||||
h.run('preflight')
|
||||
h.run('candidate')
|
||||
h.change((state) => {
|
||||
const revision = state.revisions[state.latest]
|
||||
if (defect === 'runtime') {
|
||||
revision.spec.serviceAccountName = 'wrong@test'
|
||||
}
|
||||
if (defect === 'secret') {
|
||||
revision.spec.containers[0].env[0].valueFrom.secretKeyRef.key = '8'
|
||||
}
|
||||
if (defect === 'scaling') {
|
||||
revision.metadata.annotations['autoscaling.knative.dev/maxScale'] = '3'
|
||||
}
|
||||
if (defect === 'traffic') {
|
||||
state.serving = state.latest
|
||||
}
|
||||
if (defect.startsWith('deploy-')) {
|
||||
state.failure = defect
|
||||
}
|
||||
})
|
||||
// Corrupt the recovery response after the modeled deploy while keeping real jq assertions.
|
||||
const extra = ['mode', 'image'].includes(defect)
|
||||
? `
|
||||
gcloud() {
|
||||
node "$MODEL_SCRIPT" "$@" > "$RUNNER_TEMP/out" || return $?
|
||||
if [[ "$*" == 'run services describe '* && "$*" == *'--format=json'* ]]; then
|
||||
jq '${defect === 'mode' ? '.spec.template.spec.containers[0].env += [{name:"ORCA_PUSH_MODE",value:"validation"}]' : '.spec.template.spec.containers[0].image = "wrong"'}' "$RUNNER_TEMP/out"
|
||||
else cat "$RUNNER_TEMP/out"; fi
|
||||
}`
|
||||
: ''
|
||||
h.run('restore', false, extra)
|
||||
const recorded = readFileSync(join(h.dir, 'env'), 'utf8')
|
||||
assert.match(recorded, /TEMPLATE_RECOVERY_REVISION=push-test-r123-1/)
|
||||
assert.doesNotMatch(recorded, /TEMPLATE_RESTORED=true/)
|
||||
assert.ok(h.state().peak <= 3)
|
||||
})
|
||||
)
|
||||
}
|
||||
|
||||
test('failed validation retirement blocks a fourth revision during recovery', options, () =>
|
||||
exercise((h) => {
|
||||
h.run('preflight')
|
||||
h.run('candidate')
|
||||
h.change((state) => {
|
||||
state.failure = 'delete'
|
||||
})
|
||||
h.run('activate', false)
|
||||
h.run('restore', false)
|
||||
assert.equal(h.state().peak, 3)
|
||||
assert.equal(h.state().revisions['push-test-r123-1'], undefined)
|
||||
})
|
||||
)
|
||||
|
||||
test(
|
||||
'failed retirement after public checks leaves verified serving and blocks the next run',
|
||||
options,
|
||||
() =>
|
||||
exercise((h) => {
|
||||
for (const key of ['preflight', 'candidate', 'activate', 'shift', 'public']) {
|
||||
h.run(key)
|
||||
}
|
||||
h.change((state) => {
|
||||
state.failure = 'delete'
|
||||
})
|
||||
h.run('retire', false)
|
||||
assert.equal(h.state().serving, 'push-test-a123-1')
|
||||
h.run('preflight', false)
|
||||
assert.match(workflow, /env.ROLLOUT_VERIFIED != 'true'/)
|
||||
})
|
||||
)
|
||||
|
||||
test(
|
||||
'recovery promotion failure keeps consumers for operator diagnosis and blocks new rollout',
|
||||
options,
|
||||
() =>
|
||||
exercise((h) => {
|
||||
h.run('preflight')
|
||||
h.run('candidate')
|
||||
h.run('restore')
|
||||
h.change((state) => {
|
||||
state.failure = 'public'
|
||||
})
|
||||
h.run('promoteRecovery', false)
|
||||
assert.doesNotMatch(readFileSync(join(h.dir, 'env'), 'utf8'), /RECOVERY_VERIFIED=true/)
|
||||
h.run('preflight', false)
|
||||
})
|
||||
)
|
||||
|
||||
const capability = step('Require image support for inert validation')
|
||||
for (const [label, source, ok] of [
|
||||
['old image', 'export function loadPushConfig() { return {}; }', false],
|
||||
[
|
||||
'invalid mode accepted',
|
||||
'export function loadPushConfig(env) { return { mode: env.ORCA_PUSH_MODE }; }',
|
||||
false
|
||||
],
|
||||
[
|
||||
'validation supported',
|
||||
`export function loadPushConfig(env) {
|
||||
if (env.ORCA_PUSH_MODE !== 'validation') throw new Error('invalid mode');
|
||||
return { mode: 'validation' };
|
||||
}`,
|
||||
true
|
||||
]
|
||||
]) {
|
||||
test(`pre-production image smoke: ${label}`, options, () =>
|
||||
exercise((h) => {
|
||||
const dist = join(h.dir, 'apps', 'push', 'dist')
|
||||
mkdirSync(dist, { recursive: true })
|
||||
writeFileSync(join(h.dir, 'package.json'), '{"type":"module"}')
|
||||
writeFileSync(join(dist, 'config.js'), source)
|
||||
h.run(capability, ok, 'docker() { node "${@: -3}"; }')
|
||||
})
|
||||
)
|
||||
}
|
||||
@@ -1,329 +0,0 @@
|
||||
import assert from 'node:assert/strict'
|
||||
import { readFileSync } from 'node:fs'
|
||||
import test from 'node:test'
|
||||
import {
|
||||
concurrencyBlocks,
|
||||
jobIf,
|
||||
jobs,
|
||||
LEASE_ACTION,
|
||||
leaseSteps
|
||||
} from './cloud-sql-rollout-lock-census.mjs'
|
||||
import { readRelayWorkflow, relayWorkflowFile } from './relay-repository.mjs'
|
||||
|
||||
// Why: the push gateway holds the APNs key and is the only thing standing between a paired
|
||||
// phone and a silent notification pipeline. Its deploy is a blue/green rollout against the
|
||||
// dedicated Cloud SQL instance, and each of the guarantees below is one careless edit from gone.
|
||||
const WORKFLOW = 'push-deploy.yml'
|
||||
const workflow = readRelayWorkflow(WORKFLOW)
|
||||
const deploy = () => {
|
||||
const job = jobs(workflow).find((entry) => entry.id === 'deploy')
|
||||
assert.ok(job, 'the workflow no longer declares a deploy job')
|
||||
return job
|
||||
}
|
||||
|
||||
function terraform(file) {
|
||||
return readFileSync(new URL(`../../infra/terraform/${file}`, import.meta.url), 'utf8')
|
||||
}
|
||||
|
||||
// The ordered step names; every assertion below reads positions out of this list rather than
|
||||
// restating them, so a reordering that breaks the no-traffic guarantee fails here.
|
||||
const stepNames = () => [...workflow.matchAll(/^ {6}- name: (.+)$/gm)].map((match) => match[1])
|
||||
|
||||
const indexOfStep = (name) => {
|
||||
const index = stepNames().indexOf(name)
|
||||
assert.notEqual(index, -1, `the workflow no longer has a "${name}" step`)
|
||||
return index
|
||||
}
|
||||
|
||||
test('the whole surface stays inert until the owner enables cloud operations', () => {
|
||||
const guard = jobIf(deploy().text)
|
||||
assert.ok(guard.includes("vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true'"), guard)
|
||||
assert.ok(guard.includes("github.ref == 'refs/heads/main'"), guard)
|
||||
assert.equal(jobs(workflow).length, 1, 'a second job would need its own gate')
|
||||
})
|
||||
|
||||
test('it authenticates through Workload Identity and holds no repository secret', () => {
|
||||
assert.match(workflow, /uses: google-github-actions\/auth@v2/)
|
||||
assert.match(workflow, /workload_identity_provider: \$\{\{ vars\.PRODUCTION_GCP_PUSH_DEPLOY_WORKLOAD_IDENTITY_PROVIDER \}\}/)
|
||||
assert.match(workflow, /service_account: \$\{\{ vars\.PRODUCTION_GCP_PUSH_DEPLOY_SERVICE_ACCOUNT \}\}/)
|
||||
assert.match(workflow, /environment: production/)
|
||||
for (const [, name] of workflow.matchAll(/secrets\.([A-Za-z_][A-Za-z0-9_]*)/g)) {
|
||||
assert.equal(name, 'GITHUB_TOKEN', `the workflow reads secrets.${name}`)
|
||||
}
|
||||
})
|
||||
|
||||
// Why: Terraform trusts exact workflow filenames, not a prefix. A rename here without the
|
||||
// matching tfvars-independent list entry would fail authentication at dispatch time only.
|
||||
test('Terraform trusts this exact workflow file on the production deploy provider', () => {
|
||||
assert.match(terraform('push-deploy-identity.tf'), /push-deploy\.yml@refs\/heads\/main/)
|
||||
assert.doesNotMatch(terraform('relay-github-actions.tf'), /push-deploy\.yml/)
|
||||
assert.equal(relayWorkflowFile(WORKFLOW), 'cloud-push-deploy.yml')
|
||||
})
|
||||
|
||||
test('the rollout is serialized and leases its dedicated push rollout lock', () => {
|
||||
const blocks = concurrencyBlocks(workflow)
|
||||
assert.equal(blocks.length, 1)
|
||||
assert.equal(blocks[0].group, 'production-push-rollout')
|
||||
assert.equal(blocks[0].cancelInProgress, 'false')
|
||||
const steps = leaseSteps(workflow)
|
||||
assert.equal(steps.length, 1, 'exactly one lease step, held for the whole run')
|
||||
assert.equal(steps[0].bucket, 'onorca-cloud-terraform-state')
|
||||
assert.equal(steps[0].object, 'terraform/state/push-rollout/production.lock')
|
||||
assert.equal(steps[0].release, undefined, 'release stays at its default for a single-job run')
|
||||
})
|
||||
|
||||
// Why: the ops guardrail is that a piped command only fails the step when pipefail is set, and
|
||||
// pipefail only applies under an explicit bash shell. Every multi-line body here opts in.
|
||||
test('every multi-line command runs under bash with pipefail', () => {
|
||||
const bodies = [...workflow.matchAll(/^ {8}(shell: bash\n {8})?run: \|\n((?: {10}.*\n|\n)+)/gm)]
|
||||
assert.ok(bodies.length >= 8, `only ${bodies.length} multi-line commands were found`)
|
||||
for (const match of bodies) {
|
||||
assert.ok(match[1], `a multi-line command does not declare shell: bash:\n${match[2].slice(0, 120)}`)
|
||||
assert.match(match[2], /^ {10}set -euo pipefail$/m)
|
||||
}
|
||||
})
|
||||
|
||||
test('the candidate revision takes no traffic and is addressed by its own tag', () => {
|
||||
assert.match(workflow, /gcloud run deploy "\$\{SERVICE_NAME\}"/)
|
||||
assert.match(workflow, /^ {12}--no-traffic \\$/m)
|
||||
assert.match(workflow, /--tag "\$\{tag\}"/)
|
||||
assert.match(workflow, /test "\$\{CANDIDATE_REVISION\}" != "\$\{ROLLBACK_REVISION\}"/)
|
||||
assert.ok(
|
||||
indexOfStep('Record the serving revision and require its Terraform-owned scaling') <
|
||||
indexOfStep('Deploy the candidate revision with no traffic'),
|
||||
'the rollback target must be captured before the candidate exists'
|
||||
)
|
||||
})
|
||||
|
||||
// Why: scaling is a Terraform-owned field that `lifecycle.ignore_changes` does not cover, so a
|
||||
// deploy that passed --max-instances would revert a later push_max_instances raise on every run.
|
||||
// The workflow asserts the shape instead of writing it, on the serving revision before the
|
||||
// candidate exists and on the candidate that inherits it.
|
||||
test('the deploy asserts the Terraform-owned scaling instead of mutating it', () => {
|
||||
assert.doesNotMatch(workflow, /--max-instances/, 'the deploy must not write a scaling field')
|
||||
assert.doesNotMatch(workflow, /--min-instances "/, 'the deploy must not write a scaling field')
|
||||
// The floor is the variables.tf default; production.tfvars overrides only the ceiling, down to
|
||||
// the two instances the Cloud SQL connection budget leaves room for.
|
||||
assert.match(workflow, /PUSH_MIN_INSTANCES: 1$/m)
|
||||
assert.match(workflow, /PUSH_MAX_INSTANCES: 2$/m)
|
||||
assert.match(terraform('variables.tf'), /variable "push_min_instances"[\s\S]*?default {5}= 1/)
|
||||
assert.match(terraform('environments/production.tfvars'), /^push_max_instances {9}= 2$/m)
|
||||
const gate = indexOfStep('Record the serving revision and require its Terraform-owned scaling')
|
||||
assert.ok(gate < indexOfStep('Deploy the candidate revision with no traffic'))
|
||||
assert.match(workflow, /autoscaling\.knative\.dev\/minScale/)
|
||||
assert.match(workflow, /\[\[ "\$\{floor:-0\}" -lt "\$\{PUSH_MIN_INSTANCES\}" \]\]/)
|
||||
assert.match(workflow, /test "\$\{ceiling\}" = "\$\{PUSH_MAX_INSTANCES\}"/)
|
||||
assert.match(workflow, /test "\$\{candidate_ceiling\}" = "\$\{PUSH_MAX_INSTANCES\}"/)
|
||||
})
|
||||
|
||||
// Why: the image build is not a Cloud SQL operation, and the lease is a global serialization
|
||||
// point. A build inside it blocks every relay deploy and rehome for its duration.
|
||||
test('the image is built before the rollout lease is taken', () => {
|
||||
const lease = workflow.indexOf(`- uses: ${LEASE_ACTION}`)
|
||||
assert.notEqual(lease, -1)
|
||||
const build = workflow.indexOf('- name: Build and publish the immutable gateway image')
|
||||
const deployCandidate = workflow.indexOf('- name: Deploy the candidate revision with no traffic')
|
||||
assert.ok(build < lease, 'the build must finish before the run takes the lease')
|
||||
assert.ok(lease < deployCandidate, 'the lease must still cover the deploy, probe, and shift')
|
||||
})
|
||||
|
||||
// Why: the gateway's Cloud SQL draw is instances x pool, and the root that takes the rollout
|
||||
// lease can only account for a pool it declares. Leaving it at the application default hid it.
|
||||
test('the database pool size is Terraform-owned and bounded at plan time', () => {
|
||||
const source = terraform('push-gateway.tf')
|
||||
assert.match(source, /name {2}= "ORCA_PUSH_DATABASE_POOL_MAX"/)
|
||||
assert.match(source, /value = tostring\(var\.push_database_pool_max\)/)
|
||||
assert.match(terraform('variables.tf'), /variable "push_database_pool_max"[\s\S]*?default {5}= 2/)
|
||||
const block = /resource "google_cloud_run_v2_service" "push"[\s\S]*?\n lifecycle \{([\s\S]*?)\n \}/.exec(source)
|
||||
assert.ok(block, 'the push service no longer declares a lifecycle block')
|
||||
assert.match(
|
||||
block[1],
|
||||
/var\.push_max_instances \* var\.push_database_pool_max \* 3 <= 64/,
|
||||
'instances x pool must be bounded at plan time'
|
||||
)
|
||||
assert.match(
|
||||
readFileSync(new URL('../../apps/push/src/config.ts', import.meta.url), 'utf8'),
|
||||
/ORCA_PUSH_DATABASE_POOL_MAX/,
|
||||
'the gateway must read the variable Terraform sets'
|
||||
)
|
||||
})
|
||||
|
||||
test('the candidate is probed on its own URL before any traffic moves', () => {
|
||||
const probe = indexOfStep('Probe the candidate readiness endpoint')
|
||||
assert.ok(probe > indexOfStep('Deploy the candidate revision with no traffic'))
|
||||
assert.ok(probe < indexOfStep('Shift all traffic to the verified candidate'))
|
||||
assert.match(workflow, /"\$\{CANDIDATE_URL\}\/ready"/)
|
||||
assert.match(workflow, /test "\$\{code\}" = 200/)
|
||||
assert.ok(workflow.indexOf('${CANDIDATE_URL}/ready') < workflow.indexOf('${CANDIDATE_URL}/health'))
|
||||
assert.match(workflow, /\.deliveryProtocol == 2/, 'verify the durable gateway after readiness')
|
||||
})
|
||||
|
||||
// Why: a gateway that answers /ready can still hold no usable FCM credential. The probe must be
|
||||
// validate-only, must use a token that cannot exist, and must treat a denied credential as the
|
||||
// failure. Accepting PERMISSION_DENIED would make the whole step decorative.
|
||||
test('the FCM probe is validate-only and separates a bad token from a bad credential', () => {
|
||||
const fcm = indexOfStep('Prove the runtime identity can reach FCM')
|
||||
assert.ok(fcm > indexOfStep('Probe the candidate readiness endpoint'))
|
||||
assert.ok(fcm < indexOfStep('Shift all traffic to the verified candidate'))
|
||||
assert.match(workflow, /"validate_only":true/)
|
||||
assert.match(workflow, /https:\/\/fcm\.googleapis\.com\/v1\/projects\/\$\{GCP_PROJECT_ID\}\/messages:send/)
|
||||
assert.match(workflow, /GCP_PROJECT_ID: onorca-cloud$/m)
|
||||
assert.match(workflow, /orca-push-deploy-probe-invalid-token/)
|
||||
assert.match(workflow, /test "\$\{status\}" = INVALID_ARGUMENT/)
|
||||
assert.match(workflow, /test "\$\{status\}" = PERMISSION_DENIED/)
|
||||
// Only those four answers are conclusive; a 429 or a 5xx says nothing about the credential, so
|
||||
// it is retried rather than read as either verdict. A denied credential still fails at once.
|
||||
assert.match(workflow, /for attempt in \$\(seq 1 5\); do/)
|
||||
const probe = workflow.slice(
|
||||
workflow.indexOf('- name: Prove the runtime identity can reach FCM'),
|
||||
workflow.indexOf('- name: Shift all traffic to the verified candidate')
|
||||
)
|
||||
assert.match(probe, /for attempt in \$\(seq 1 5\); do/)
|
||||
assert.match(probe, /test "\$\{code\}" = 401 \|\| test "\$\{code\}" = 403; then\n {14}break/)
|
||||
assert.match(
|
||||
workflow,
|
||||
/--impersonate-service-account "\$\{PUSH_RUNTIME_SERVICE_ACCOUNT\}"/,
|
||||
'the probe must exercise the runtime credential, not the deploy identity'
|
||||
)
|
||||
// Why: that token reads the Apple signing key. Masking it means a later `set -x` or a
|
||||
// debug re-run cannot print it into a public log.
|
||||
assert.match(
|
||||
probe,
|
||||
/test -n "\$\{token\}"\n {10}echo "::add-mask::\$\{token\}"/,
|
||||
'the impersonated token must be masked before anything else runs'
|
||||
)
|
||||
assert.match(workflow, /PUSH_RUNTIME_SERVICE_ACCOUNT: orca-cloud-push@onorca-cloud\.iam\.gserviceaccount\.com/)
|
||||
})
|
||||
|
||||
// Why: a deploy ends with traffic pinned to an exact revision, and a rollback pins it to the
|
||||
// previous one. Terraform reverting the service to 100% LATEST would undo either silently.
|
||||
test('Terraform does not own the image or the traffic split', () => {
|
||||
const source = terraform('push-gateway.tf')
|
||||
const block = /resource "google_cloud_run_v2_service" "push"[\s\S]*?\n lifecycle \{([\s\S]*?)\n \}/.exec(source)
|
||||
assert.ok(block, 'the push service no longer declares a lifecycle block')
|
||||
assert.match(block[1], /template\[0\]\.containers\[0\]\.image/)
|
||||
assert.match(block[1], /^\s*traffic$/m)
|
||||
})
|
||||
|
||||
test('impersonating the runtime identity is a Terraform-declared grant', () => {
|
||||
const source = terraform('push-gateway.tf')
|
||||
assert.match(source, /resource "google_service_account_iam_member" "github_production_push_runtime_token_creator"/)
|
||||
assert.match(source, /role\s+= "roles\/iam\.serviceAccountTokenCreator"/)
|
||||
assert.match(source, /resource "google_cloud_run_v2_service_iam_member" "github_production_push_developer"/)
|
||||
})
|
||||
|
||||
test('the traffic shift is all-or-nothing and is verified after the fact', () => {
|
||||
const shift = indexOfStep('Shift all traffic to the verified candidate')
|
||||
assert.match(workflow, /gcloud run services update-traffic "\$\{SERVICE_NAME\}"/)
|
||||
assert.match(workflow, /--to-revisions "\$\{CANDIDATE_REVISION\}=100"/)
|
||||
assert.match(workflow, /test "\$\{serving\}" = "\$\{CANDIDATE_REVISION\}"/)
|
||||
assert.ok(shift < indexOfStep('Verify the public origin after the shift'))
|
||||
assert.match(workflow, /PUSH_ORIGIN: https:\/\/push\.onorca\.dev/)
|
||||
assert.match(workflow, /"\$\{PUSH_ORIGIN\}\/ready"/)
|
||||
})
|
||||
|
||||
// Why: the origin can lag the traffic move by seconds, and a single unlucky curl would otherwise
|
||||
// roll a healthy deploy back. It retries on the same schedule as the candidate probe.
|
||||
test('the post-shift origin check retries like the candidate probe', () => {
|
||||
const check = workflow.slice(
|
||||
workflow.indexOf('- name: Verify the public origin after the shift'),
|
||||
workflow.indexOf('- name: Roll traffic back to the previous revision')
|
||||
)
|
||||
assert.match(check, /for attempt in \$\(seq 1 30\); do/)
|
||||
assert.match(check, /sleep 5/)
|
||||
assert.match(check, /test "\$\{code\}" = 200/)
|
||||
})
|
||||
|
||||
// Why: the summary carries the rollback target. Writing it after the origin check meant the one
|
||||
// run that needed it, the run whose check failed, was the one run that never got it.
|
||||
test('the summary is written before anything that can fail after the shift', () => {
|
||||
const summary = indexOfStep('Publish the rollout summary')
|
||||
assert.ok(summary > indexOfStep('Shift all traffic to the verified candidate'))
|
||||
assert.ok(summary < indexOfStep('Verify the public origin after the shift'))
|
||||
assert.match(workflow, /Known-good image:/)
|
||||
assert.match(workflow, /GITHUB_STEP_SUMMARY/)
|
||||
})
|
||||
|
||||
// Why: everything after the shift runs with production on the candidate, so a failure there is a
|
||||
// live gateway that has to go back. The marker is what separates that case from a failure before
|
||||
// the shift, where production never moved and the candidate is the thing to clean up.
|
||||
test('a failure after the shift rolls production back automatically', () => {
|
||||
const rollback = indexOfStep('Roll traffic back to the previous revision')
|
||||
assert.ok(rollback > indexOfStep('Verify the public origin after the shift'))
|
||||
assert.match(workflow, /echo "TRAFFIC_SHIFTED=true" >> "\$\{GITHUB_ENV\}"/)
|
||||
const shift = workflow.indexOf('- name: Shift all traffic to the verified candidate')
|
||||
assert.ok(
|
||||
workflow.indexOf('echo "TRAFFIC_SHIFTED=true"') > shift,
|
||||
'the success marker follows the shift step'
|
||||
)
|
||||
const body = workflow.slice(
|
||||
workflow.indexOf('- name: Roll traffic back to the previous revision'),
|
||||
workflow.indexOf('- name: Delete the rejected candidate revision')
|
||||
)
|
||||
assert.match(
|
||||
body,
|
||||
/if: \$\{\{ \(failure\(\) \|\| cancelled\(\)\) && env\.TRAFFIC_SHIFT_ATTEMPTED == 'true' && env\.ROLLOUT_VERIFIED != 'true' \}\}/,
|
||||
'the rollback must be conditioned on both failure and the shift marker'
|
||||
)
|
||||
assert.match(body, /test -n "\$\{ROLLBACK_REVISION:-\}"/)
|
||||
assert.match(body, /--to-revisions "\$\{ROLLBACK_REVISION\}=100"/)
|
||||
assert.match(body, /test "\$\{serving\}" = "\$\{ROLLBACK_REVISION\}"/)
|
||||
assert.match(body, /GITHUB_STEP_SUMMARY/, 'the rollback must be reported in the summary')
|
||||
})
|
||||
|
||||
// Why: a candidate that never took traffic still holds a warm instance and a Cloud SQL pool. Its
|
||||
// tag comes off first, because Cloud Run refuses to delete a revision a traffic target names.
|
||||
test('verified recovery authorizes rejected candidate deletion', () => {
|
||||
const body = workflow.slice(
|
||||
workflow.indexOf('- name: Delete the rejected candidate revision'),
|
||||
workflow.indexOf('- name: Drop the candidate traffic tag')
|
||||
)
|
||||
assert.match(
|
||||
body,
|
||||
/env\.RECOVERY_VERIFIED == 'true'/,
|
||||
'cleanup must wait for verified recovery traffic and public checks'
|
||||
)
|
||||
assert.match(body, /if test -z "\$\{CANDIDATE_REVISION:-\}"; then/)
|
||||
assert.ok(
|
||||
body.indexOf('--remove-tags') < body.indexOf('gcloud run revisions delete'),
|
||||
'the tag must come off before the revision is deleted'
|
||||
)
|
||||
assert.match(body, /echo "CANDIDATE_TAG=" >> "\$\{GITHUB_ENV\}"/)
|
||||
})
|
||||
|
||||
test('the run always drops its traffic tag', () => {
|
||||
const cleanup = indexOfStep('Drop the candidate traffic tag')
|
||||
assert.equal(cleanup, stepNames().length - 1, 'tag cleanup must be the last step')
|
||||
assert.match(workflow, /--remove-tags "\$\{CANDIDATE_TAG\}"/)
|
||||
const body = workflow.slice(workflow.indexOf('- name: Drop the candidate traffic tag'))
|
||||
assert.match(body, /if: always\(\)/)
|
||||
assert.match(body, /test -n "\$\{CANDIDATE_TAG:-\}" \|\| exit 0/)
|
||||
})
|
||||
|
||||
test('push credentials cannot assume the shared Relay deploy identity', () => {
|
||||
const source = terraform('push-deploy-identity.tf')
|
||||
assert.match(source, /"attribute.push_deploy"\s*=\s*"'production'"/)
|
||||
assert.doesNotMatch(source, /"attribute.repository"\s*=/)
|
||||
assert.match(source, /attribute\.push_deploy\/production/)
|
||||
assert.doesNotMatch(workflow, /PRODUCTION_GCP_RELAY_DEPLOY_/)
|
||||
assert.doesNotMatch(terraform('push-gateway.tf'), /member\s*=\s*local\.relay_github_deploy_service_account_member/)
|
||||
})
|
||||
|
||||
test('push has only a dedicated database attachment and a narrowly scoped deployment lease', () => {
|
||||
const service = terraform('push-gateway.tf')
|
||||
const database = terraform('push-dedicated-database.tf')
|
||||
assert.match(service, /instances = \[google_sql_database_instance\.push_dedicated\[0\]\.connection_name\]/)
|
||||
assert.match(service, /secret\s*= google_secret_manager_secret\.push_dedicated_database_url\[0\]\.secret_id/)
|
||||
assert.match(service, /version = google_secret_manager_secret_version\.push_dedicated_database_url\[0\]\.version/)
|
||||
assert.doesNotMatch(service + database, /push_dedicated_database_(?:active|enabled)|local\.relay_database_connection_name|resource "google_sql_database" "push"/)
|
||||
assert.match(database, /tier\s*= "db-custom-2-7680"/)
|
||||
assert.match(database, /availability_type = "REGIONAL"/)
|
||||
assert.match(database, /deletion_protection\s*= true/)
|
||||
assert.match(database, /deletion_protection_enabled = true/)
|
||||
const identity = terraform('push-deploy-identity.tf')
|
||||
const lease = identity.match(/resource "google_storage_bucket_iam_member" "github_push_rollout_lease" \{([\s\S]*?)\n\}/)?.[1]
|
||||
assert.ok(lease)
|
||||
assert.match(lease, /member = local\.push_deploy_member/)
|
||||
assert.match(lease, /role\s*= "roles\/storage.objectAdmin"/)
|
||||
assert.match(lease, /resource.name == 'projects\/_\/buckets\/\$\{var.project_id\}-terraform-state\/objects\/terraform\/state\/push-rollout\/production.lock'/)
|
||||
})
|
||||
@@ -1,31 +0,0 @@
|
||||
import { readFileSync } from 'node:fs'
|
||||
import { pathToFileURL } from 'node:url'
|
||||
|
||||
const CAPACITY_IDENTITY_NAME = 'ORCA_RELAY_CAPACITY_SERVICE_ACCOUNT'
|
||||
|
||||
export function readProductionCapacityIdentity(revision) {
|
||||
const env = revision?.spec?.containers?.[0]?.env
|
||||
if (!Array.isArray(env)) throw new Error('director revision environment is missing')
|
||||
const matches = env.filter((entry) => entry?.name === CAPACITY_IDENTITY_NAME)
|
||||
if (matches.length === 0) return null
|
||||
if (matches.length !== 1) throw new Error('duplicate capacity identity')
|
||||
const value = matches[0]?.value
|
||||
if (typeof value !== 'string' || value.length === 0) {
|
||||
throw new Error('capacity identity is not a literal string')
|
||||
}
|
||||
return value
|
||||
}
|
||||
|
||||
export function main() {
|
||||
const revision = JSON.parse(readFileSync(0, 'utf8'))
|
||||
process.stdout.write(`${JSON.stringify(readProductionCapacityIdentity(revision))}\n`)
|
||||
}
|
||||
|
||||
if (import.meta.url === pathToFileURL(process.argv[1]).href) {
|
||||
try {
|
||||
main()
|
||||
} catch (error) {
|
||||
process.stderr.write(`${error instanceof Error ? error.message : String(error)}\n`)
|
||||
process.exitCode = 1
|
||||
}
|
||||
}
|
||||
@@ -1,44 +0,0 @@
|
||||
import assert from 'node:assert/strict'
|
||||
import test from 'node:test'
|
||||
import { readProductionCapacityIdentity } from './read-relay-production-capacity-identity.mjs'
|
||||
|
||||
function revision(env) {
|
||||
return { spec: { containers: [{ env }] } }
|
||||
}
|
||||
|
||||
test('reads absent, exact, and foreign literal capacity identities', () => {
|
||||
assert.equal(readProductionCapacityIdentity(revision([])), null)
|
||||
assert.equal(
|
||||
readProductionCapacityIdentity(revision([
|
||||
{ name: 'ORCA_RELAY_CAPACITY_SERVICE_ACCOUNT', value: 'capacity@example.test' }
|
||||
])),
|
||||
'capacity@example.test'
|
||||
)
|
||||
assert.equal(
|
||||
readProductionCapacityIdentity(revision([
|
||||
{ name: 'ORCA_RELAY_CAPACITY_SERVICE_ACCOUNT', value: 'foreign@example.test' }
|
||||
])),
|
||||
'foreign@example.test'
|
||||
)
|
||||
})
|
||||
|
||||
test('rejects malformed or duplicate capacity identity entries', () => {
|
||||
for (const entry of [
|
||||
{ name: 'ORCA_RELAY_CAPACITY_SERVICE_ACCOUNT', value: null },
|
||||
{ name: 'ORCA_RELAY_CAPACITY_SERVICE_ACCOUNT', value: '' },
|
||||
{ name: 'ORCA_RELAY_CAPACITY_SERVICE_ACCOUNT', valueSource: { secretKeyRef: {} } }
|
||||
]) {
|
||||
assert.throws(
|
||||
() => readProductionCapacityIdentity(revision([entry])),
|
||||
/not a literal string/
|
||||
)
|
||||
}
|
||||
assert.throws(
|
||||
() => readProductionCapacityIdentity(revision([
|
||||
{ name: 'ORCA_RELAY_CAPACITY_SERVICE_ACCOUNT', value: 'one@example.test' },
|
||||
{ name: 'ORCA_RELAY_CAPACITY_SERVICE_ACCOUNT', value: 'two@example.test' }
|
||||
])),
|
||||
/duplicate capacity identity/
|
||||
)
|
||||
assert.throws(() => readProductionCapacityIdentity({}), /environment is missing/)
|
||||
})
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user