mirror of
https://github.com/stablyai/orca.git
synced 2026-11-01 17:47:40 +00:00
* feat(relay): watch a reserve cell and flip it back on the first step 5 stop rule A manual, inert-until-dispatched workflow samples one reserve cell every minute for up to 300 minutes against the step 5 §9 stop rules, logs one JSON line per sample, and on the first trip (or N unreadable samples in a row) runs the flag tool's own admitMode=db write under the flag workflow's concurrency group. dry-run logs the flag tool's plan instead. Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * fix(relay): keep the reserve watch's flip back out of the flag group, and baseline 503s before the flip - The flip back holds no concurrency group: GitHub keeps one pending run per group and a newer one cancels it. The flag tool's CAS orders it instead; a generation conflict is re-read and retried (5 tries, backoff) while the switch file still says reserve. - The 503 background comes from the 10 minutes before a required flipped-at, or an explicit value; neither is refused. - A failed or cancelled write-mode watch flips back too. An unreachable cell fails with the exact break-glass inputs. - Missing 503 data (empty series, unpublished minute, unparseable time) is unverifiable. - Seat-behind-row key adds the director instance and cell; six segments share one command. Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * fix(relay): read the reserve watch's 503 publishing front from every response code A quiet fleet has no 503 series, so the watch read it as unverifiable and flipped a healthy cell back within minutes. The 503 read now groups request_count by response code in one query; the newest minute any code has a point for is how far the series is published, so a 503 minute at or before it with no point is zero and minutes past it are left out. Also: the flip-back job's steps run on always(), flipped-at ages against the watch's start in later segments, a pending pre-flip background no longer adds to the unverifiable streak, the flip back retries runtime-status before calling for break glass, and the concurrency rationale lives once in the docs. Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * docs(relay): state what a cancel during the flip-back lease wait can leave, and the 503 publish-skew assumption Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * fix(relay): fail a segment closed on a corrupt watch state file, and keep no checkout token on disk Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * fix(relay): alarm on a failed flip back, refuse a watch under a rollout lease, wait for reserve to apply, and control the duplicate read - A flip back whose write or plan never finished (no ID token, a failed login, a silent cell) raises an ::error:: annotation and a FLIP BACK FAILED summary, as the clock-skew monitor does. - The first segment reads the Cloud SQL rollout lease through the lease action's own reader and refuses to start while any run holds it, or when it cannot be read. - A cell not yet reporting reserve is waited on for up to 10 minutes, judging nothing, before the watch ends refused without a write. - The duplicate-seat read takes the per-minute ledger summary in the same query and window; with no fresh summary in it the sample is unverifiable, not zero duplicates. Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * fix(relay): page on a refused reserve watch or a failed flip back A refused watch can leave a cell in reserve and unwatched, the same as a flip back that did not land. Both now raise an escaped ::error:: annotation through one `alarm` command, and in write mode write orca_relay_reserve_watch_alarm with gcloud logging write. A new log-based metric and alert policy page on any such line, per cell, through relay_alert_notification_channels, and the deploy identity gets roles/logging.logWriter in production to write it. Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * fix(relay): name the log-writer risk, check the alarm's project, and say a failed login cannot page Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * docs(relay): note a force-cancel can skip the reserve-watch flip-back write
582 lines
27 KiB
JavaScript
582 lines
27 KiB
JavaScript
#!/usr/bin/env node
|
|
// Watches one cell already in reserve mode and, on the first stop rule it trips, flips it back to
|
|
// db through the flag tool's own write (CAS, read-back, re-registration wait). Step 5 design §9.
|
|
//
|
|
// watch: one segment of the watch (the workflow re-authenticates between segments, since an
|
|
// ID token lasts an hour). State carries over in --state-file. Writes nothing.
|
|
// outcome: the job's state and verdict, from the state file.
|
|
// flip-back: the flag tool's admitMode=db write (or its dry run), then exits non-zero with the
|
|
// watch's verdict. Why it holds no flag-workflow group: docs/relay-workflows.md.
|
|
// alarm: a refused watch or a flip back that did not land: an error annotation, and in write
|
|
// mode a log line the reserve-watch alert pages on.
|
|
import { execFile } from 'node:child_process'
|
|
import { appendFile, readFile, writeFile } from 'node:fs/promises'
|
|
import { parseArgs, promisify } from 'node:util'
|
|
import { pathToFileURL } from 'node:url'
|
|
import {
|
|
BREAK_GLASS_CONFIRMATION,
|
|
operateCellFlags,
|
|
parseCellFlagsRequest,
|
|
readCurrentObject,
|
|
readRuntime,
|
|
WRITE_CONFIRMATION
|
|
} from './operate-relay-cell-flags.mjs'
|
|
import { readDirectorRequestsByCodePerMinute, readLogEntries } from './relay-same-cap-shadow-gate.mjs'
|
|
import { minutesOf } from './relay-same-cap-shadow-gate-verdict.mjs'
|
|
import { PRODUCTION_LEASE } from './cloud-sql-rollout-lock-census.mjs'
|
|
import { RELAY_REPOSITORY_ROOT } from './relay-repository.mjs'
|
|
import {
|
|
evaluateWatchSample,
|
|
initialWatchState,
|
|
measuredBackground,
|
|
RESERVE_WATCH_RULES
|
|
} from './relay-reserve-watch-rules.mjs'
|
|
|
|
const execFileAsync = promisify(execFile)
|
|
const MINUTE_MS = 60_000
|
|
export const SAMPLE_INTERVAL_MS = MINUTE_MS
|
|
export const MAX_WATCH_MINUTES = 300
|
|
// Below the hour an ID token lasts, with room for the segment's last sample.
|
|
export const MAX_SEGMENT_MINUTES = 55
|
|
const METRIC_DELAY_MS = RESERVE_WATCH_RULES.director503.publishDelayMs
|
|
const BACKGROUND_MINUTES = 10
|
|
// A flipped-at older than this is a typo or a stale dispatch, not this flip.
|
|
const MAX_FLIPPED_AT_AGE_MS = 24 * 60 * MINUTE_MS
|
|
export const FLIP_BACK_ATTEMPTS = 5
|
|
const FLIP_BACK_BACKOFF_MS = 2_000
|
|
// One slow runtime-status reply is not a dead cell: retried this often before break glass.
|
|
export const FLIP_BACK_RUNTIME_ATTEMPTS = 4
|
|
const SAMPLE_READ_BUDGET_MS = 40_000
|
|
const LOG_ENTRY_LIMIT = 1_000
|
|
|
|
function integerIn(name, raw, min, max) {
|
|
const value = Number(raw)
|
|
if (!/^[0-9]+$/.test(raw ?? '') || value < min || value > max) {
|
|
throw new Error(`--${name} must be an integer from ${min} to ${max}`)
|
|
}
|
|
return value
|
|
}
|
|
|
|
// The flip back's request. Its generation is a placeholder: the flip back re-reads the current one.
|
|
export function dbFlipRequest(values) {
|
|
return parseCellFlagsRequest({ ...values, set: 'admitMode=db', 'expected-generation': '0' })
|
|
}
|
|
|
|
// `now` is the watch's start: a later segment of a long watch must not age flipped-at out.
|
|
function parseFlippedAt(raw, now) {
|
|
if (raw === '') return null
|
|
const at = Date.parse(raw)
|
|
if (!/^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}(:\d{2}(\.\d{1,3})?)?(Z|[+-]\d{2}:\d{2})$/.test(raw) || !Number.isFinite(at)) {
|
|
throw new Error('--flipped-at must be an ISO time with a zone, e.g. 2026-10-14T12:00:00Z')
|
|
}
|
|
if (at > now) throw new Error('--flipped-at is in the future')
|
|
if (at < now - MAX_FLIPPED_AT_AGE_MS) throw new Error('--flipped-at is more than 24 hours ago')
|
|
return at
|
|
}
|
|
|
|
export function parseWatchArguments(values, now = Date.now()) {
|
|
// The flag tool's own checks, so the watch is of exactly the cell the flip back would write;
|
|
// a write-mode watch refuses a missing confirmation now, not hours later.
|
|
const request = dbFlipRequest(values)
|
|
if (!/^[a-z][a-z0-9-]{0,62}$/.test(values['director-service'] ?? '')) {
|
|
throw new Error('--director-service is not a Cloud Run service name')
|
|
}
|
|
const background = values['background-503-per-minute'] ?? ''
|
|
if (background !== '' && !/^(0|[1-9][0-9]{0,6})(\.[0-9]+)?$/.test(background)) {
|
|
throw new Error('--background-503-per-minute must be empty (measure it) or a non-negative number')
|
|
}
|
|
// The background must predate the flip: a watch starts after it, so its own past is not one.
|
|
const flippedAt = parseFlippedAt(values['flipped-at'] ?? '', now)
|
|
if (background === '' && flippedAt === null) {
|
|
throw new Error('--flipped-at (the background is the 10 minutes before it) or --background-503-per-minute is required')
|
|
}
|
|
if (!values['state-file']) throw new Error('--state-file is required')
|
|
return {
|
|
request,
|
|
directorService: values['director-service'],
|
|
durationMinutes: integerIn('duration-minutes', values['duration-minutes'], 5, MAX_WATCH_MINUTES),
|
|
segmentMinutes: integerIn('segment-minutes', values['segment-minutes'], 1, MAX_SEGMENT_MINUTES),
|
|
unverifiableLimit: integerIn('unverifiable-limit', values['unverifiable-limit'], 1, 30),
|
|
background503PerMinute: background === '' ? null : Number(background),
|
|
flippedAt,
|
|
stateFile: values['state-file'],
|
|
githubOutput: values['github-output'] || null
|
|
}
|
|
}
|
|
|
|
const failedRead = (error) => ({ ok: false, error: error instanceof Error ? error.message : String(error) })
|
|
|
|
export async function readCellSample(fetchImpl, request, idToken, wait) {
|
|
try {
|
|
const runtime = await readRuntime(fetchImpl, request, idToken, wait)
|
|
if (runtime.role !== 'cell' || runtime.cellId !== request.cellId) {
|
|
return failedRead(`origin answers as ${runtime.role}/${runtime.cellId}`)
|
|
}
|
|
if (!runtime.flagsApplied) return failedRead('runtime-status reports no flagsApplied')
|
|
const controls = runtime.runtime?.controls
|
|
if (!Number.isInteger(controls)) return failedRead('runtime-status reports no control count')
|
|
return {
|
|
ok: true,
|
|
admitModeConfigured: runtime.flagsApplied.flags?.admitMode ?? 'db',
|
|
admitModeEffective: runtime.admitModeEffective,
|
|
draining: runtime.draining === true,
|
|
controls,
|
|
enforcedUnits: runtime.runtime?.enforcedConnectionUnits,
|
|
hardCap: runtime.connectionCapacity?.hardCap
|
|
}
|
|
} catch (error) {
|
|
return failedRead(error)
|
|
}
|
|
}
|
|
|
|
const minuteFloor = (at) => Math.floor(at / MINUTE_MS) * MINUTE_MS
|
|
const failed503Read = (error) => ({ ok: false, error, perMinute: {}, minutes: [], publishedThrough: null })
|
|
|
|
// The `minutes` whole minutes that end at the minute boundary `endedAt`: the 503 counts, and the
|
|
// newest minute any response code has a point for. The director always serves something, so that
|
|
// front is how far the 503 series is published; no front at all is no evidence. A code publishing
|
|
// late only delays a trip: a later read sees its points.
|
|
export async function readDirector503Minutes(reader, { projectId, directorService, endedAt, minutes }) {
|
|
const span = { startedAt: new Date(endedAt - minutes * MINUTE_MS), endedAt: new Date(endedAt) }
|
|
let read
|
|
try {
|
|
read = await readDirectorRequestsByCodePerMinute(reader, { config: { projectId, directorService }, span })
|
|
} catch (error) {
|
|
// A point with an unparseable end time has no minute.
|
|
return failed503Read(`director 503 metric unreadable: ${error instanceof Error ? error.message : error}`)
|
|
}
|
|
if (read.failed) return failed503Read('director 503 metric read failed')
|
|
const published = Object.values(read.byCode).flatMap((perMinute) => Object.keys(perMinute)).sort()
|
|
return {
|
|
ok: true,
|
|
perMinute: read.byCode['503'] ?? {},
|
|
minutes: minutesOf(span),
|
|
publishedThrough: published.at(-1) ?? null
|
|
}
|
|
}
|
|
|
|
const LEDGER_SUMMARY_EVENT = 'orca_relay_reserve_ledger_summary'
|
|
const SEAT_BEHIND_ROW_EVENT = 'orca_relay_reserve_ledger_seat_behind_row'
|
|
|
|
function directorLogFilter(directorService, eventClause, sinceAt) {
|
|
return [
|
|
'resource.type="cloud_run_revision"',
|
|
`resource.labels.service_name="${directorService}"`,
|
|
eventClause,
|
|
`timestamp>="${new Date(sinceAt).toISOString()}"`
|
|
].join(' AND ')
|
|
}
|
|
|
|
// Newest summary per director instance; queued plus dropped rows are the ones not in Postgres.
|
|
export async function readLedgerSample(reader, { directorService, at }) {
|
|
const read = await readLogEntries(reader, {
|
|
filter: directorLogFilter(directorService, `jsonPayload.event="${LEDGER_SUMMARY_EVENT}"`, at - RESERVE_WATCH_RULES.ledgerSummaryMaxAgeMs),
|
|
projection: 'json(timestamp,labels.instanceId,jsonPayload.pending,jsonPayload.dropped)',
|
|
limit: LOG_ENTRY_LIMIT
|
|
})
|
|
if (read.failed) return failedRead(read.error)
|
|
if (read.entries.length >= LOG_ENTRY_LIMIT) return failedRead('ledger summary read truncated')
|
|
if (read.entries.length === 0) return failedRead('no placing director logged a ledger summary')
|
|
if (read.entries.some((entry) => !Number.isFinite(Date.parse(entry.timestamp)))) {
|
|
return failedRead('ledger summary with an unparseable timestamp')
|
|
}
|
|
const newest = new Map()
|
|
for (const entry of read.entries) {
|
|
const instance = entry.labels?.instanceId ?? 'unknown'
|
|
const previous = newest.get(instance)
|
|
if (!previous || Date.parse(entry.timestamp) > Date.parse(previous.timestamp)) newest.set(instance, entry)
|
|
}
|
|
let unwrittenRows = 0
|
|
for (const entry of newest.values()) {
|
|
const pending = Number(entry.jsonPayload?.pending)
|
|
const dropped = Number(entry.jsonPayload?.dropped)
|
|
if (!Number.isFinite(pending) || !Number.isFinite(dropped)) return failedRead('ledger summary without counts')
|
|
unwrittenRows += pending + dropped
|
|
}
|
|
return { ok: true, unwrittenRows, directors: newest.size }
|
|
}
|
|
|
|
// The same read also takes the ledger summary every placing director logs each minute, as a
|
|
// positive control: with no fresh summary in it, no seat-behind-row line is blindness, not zero.
|
|
export async function readSeatBehindRowSample(reader, { directorService, cellId, at }) {
|
|
const read = await readLogEntries(reader, {
|
|
filter: directorLogFilter(
|
|
directorService,
|
|
`((jsonPayload.event="${SEAT_BEHIND_ROW_EVENT}" AND jsonPayload.cellId="${cellId}") OR jsonPayload.event="${LEDGER_SUMMARY_EVENT}")`,
|
|
at - RESERVE_WATCH_RULES.duplicateSeat.lookbackMs
|
|
),
|
|
projection: 'json(timestamp,labels.instanceId,jsonPayload.event,jsonPayload.epoch,jsonPayload.rowEpoch)',
|
|
limit: LOG_ENTRY_LIMIT
|
|
})
|
|
if (read.failed) return failedRead(read.error)
|
|
if (read.entries.length >= LOG_ENTRY_LIMIT) return failedRead('seat-behind-row read truncated')
|
|
const reports = []
|
|
let controlSeen = false
|
|
for (const entry of read.entries) {
|
|
const reportedAt = Date.parse(entry.timestamp)
|
|
if (!Number.isFinite(reportedAt)) return failedRead('seat-behind-row line with an unparseable timestamp')
|
|
if (entry.jsonPayload?.event === LEDGER_SUMMARY_EVENT) {
|
|
controlSeen ||= reportedAt >= at - RESERVE_WATCH_RULES.ledgerSummaryMaxAgeMs
|
|
continue
|
|
}
|
|
const { epoch, rowEpoch } = entry.jsonPayload ?? {}
|
|
if (epoch === undefined || rowEpoch === undefined) return failedRead('seat-behind-row line without epochs')
|
|
// Everything the line carries but its time, which differs on every report of one seat. A
|
|
// proxy for the host: two hosts sharing it read as one seat, so it can only over-count.
|
|
reports.push({ at: reportedAt, key: `${entry.labels?.instanceId ?? 'unknown'}/${cellId}/${epoch}:${rowEpoch}` })
|
|
}
|
|
if (!controlSeen) return failedRead('seat-behind-row read holds no fresh ledger summary (positive control)')
|
|
return { ok: true, reports }
|
|
}
|
|
|
|
function createReader(dependencies, projectId) {
|
|
const now = dependencies.now
|
|
return {
|
|
runGcloud: dependencies.runGcloud,
|
|
fetch: dependencies.fetchImpl,
|
|
retryDelayMs: dependencies.retryDelayMs ?? 5_000,
|
|
readTimeoutMs: 20_000,
|
|
now,
|
|
// Every sample gets its own budget, so a slow read costs one sample, not the watch.
|
|
deadlineAt: now() + SAMPLE_READ_BUDGET_MS,
|
|
projectId
|
|
}
|
|
}
|
|
|
|
/**
|
|
* Same-cap and capacity rolls hold the Cloud SQL rollout lease for the whole run; it names no
|
|
* cell, so any live holder refuses the watch's start. Read only: holding it for hours would block
|
|
* every rollout. An unreadable lease refuses too (fail closed, as the lease action does).
|
|
*/
|
|
export async function rolloutLeaseRefusal(readRolloutLease, now) {
|
|
try {
|
|
const existing = await readRolloutLease()
|
|
if (!existing || existing.record.expires_at <= now) return null
|
|
return (
|
|
`a rollout holds the Cloud SQL rollout lease until ${new Date(existing.record.expires_at).toISOString()} ` +
|
|
`(${existing.holder}); start the watch after it ends`
|
|
)
|
|
} catch (error) {
|
|
return `the Cloud SQL rollout lease is unreadable (${error instanceof Error ? error.message : error}); refusing to start`
|
|
}
|
|
}
|
|
|
|
// The lease action's own reader, resolved from the repository root because this tree's depth
|
|
// differs between the two repositories' copies.
|
|
async function readProductionRolloutLease() {
|
|
const action = new URL('.github/actions/cloud-sql-rollout-lease/', RELAY_REPOSITORY_ROOT)
|
|
const [{ CloudSqlRolloutLease, describeHolder }, { createAccessTokenSource }] = await Promise.all([
|
|
import(new URL('storage-lease.mjs', action).href),
|
|
import(new URL('gcloud-access-token.mjs', action).href)
|
|
])
|
|
const lease = new CloudSqlRolloutLease({
|
|
bucket: PRODUCTION_LEASE.bucket,
|
|
objectName: PRODUCTION_LEASE.object,
|
|
accessToken: createAccessTokenSource()
|
|
})
|
|
const existing = await lease.read()
|
|
return existing && { record: existing.record, holder: describeHolder(existing.record) }
|
|
}
|
|
|
|
// Only a missing file is a fresh watch. Anything else that is not a state fails the segment, which
|
|
// the outcome step reads as failed and flips back: a corrupt file must not restart or refuse the watch.
|
|
export async function loadState(path) {
|
|
const text = await readFile(path, 'utf8').catch((error) => (error?.code === 'ENOENT' ? null : Promise.reject(error)))
|
|
if (text === null) return null
|
|
const state = JSON.parse(text)
|
|
const isObject = state !== null && typeof state === 'object' && !Array.isArray(state)
|
|
if (!isObject || typeof state.state !== 'string') throw new Error(`state file ${path} holds no watch state`)
|
|
if (state.state === 'watching' && (typeof state.cellId !== 'string' || !Number.isFinite(state.startedAt))) {
|
|
throw new Error(`state file ${path} has no cell or start`)
|
|
}
|
|
return state
|
|
}
|
|
|
|
// A segment's config. Later segments check flipped-at against the watch's start; bad inputs before
|
|
// the first sample are refused, so no flip back follows a typo.
|
|
export async function parseWatchSegment(values, now = Date.now()) {
|
|
const existing = values['state-file'] ? await loadState(values['state-file']) : null
|
|
try {
|
|
return parseWatchArguments(values, Number.isFinite(existing?.startedAt) ? existing.startedAt : now)
|
|
} catch (error) {
|
|
if (values['state-file'] && existing === null) {
|
|
await writeFile(values['state-file'], JSON.stringify({ state: 'refused', verdict: error.message }))
|
|
}
|
|
throw error
|
|
}
|
|
}
|
|
|
|
export async function watchSegment(config, dependencies) {
|
|
const now = dependencies.now ?? Date.now
|
|
const sleep = dependencies.sleep ?? ((ms) => new Promise((resolve) => setTimeout(resolve, ms)))
|
|
const log = dependencies.log ?? console.log
|
|
const deps = { ...dependencies, now, sleep, fetchImpl: dependencies.fetchImpl ?? fetch }
|
|
const { request, directorService } = config
|
|
let state = await loadState(config.stateFile)
|
|
if (state === null) {
|
|
const refusal = await rolloutLeaseRefusal(dependencies.readRolloutLease, now())
|
|
if (refusal) {
|
|
state = { state: 'refused', verdict: refusal }
|
|
await writeFile(config.stateFile, JSON.stringify(state))
|
|
if (config.githubOutput) await writeOutcome(config.githubOutput, state)
|
|
return state
|
|
}
|
|
state = initialWatchState({
|
|
cellId: request.cellId,
|
|
startedAt: now(),
|
|
durationMinutes: config.durationMinutes,
|
|
background503PerMinute: config.background503PerMinute,
|
|
flippedAt: config.flippedAt
|
|
})
|
|
}
|
|
if (state.cellId !== request.cellId) throw new Error(`state file is for ${state.cellId}, not ${request.cellId}`)
|
|
const segmentEndsAt = now() + config.segmentMinutes * MINUTE_MS
|
|
while (state.state === 'watching' && now() < segmentEndsAt) {
|
|
const at = now()
|
|
const baselineEndsAt = state.flippedAt === null ? null : minuteFloor(state.flippedAt)
|
|
if (!state.background && baselineEndsAt !== null && at >= baselineEndsAt + METRIC_DELAY_MS) {
|
|
// The minutes just before the flip, once published; retried every sample until one reads.
|
|
const read = await readDirector503Minutes(createReader(deps, request.projectId), {
|
|
projectId: request.projectId,
|
|
directorService,
|
|
endedAt: baselineEndsAt,
|
|
minutes: BACKGROUND_MINUTES
|
|
})
|
|
state = { ...state, background: read.ok ? measuredBackground(read.perMinute, read.minutes, read.publishedThrough) : null }
|
|
}
|
|
const reader = createReader(deps, request.projectId)
|
|
// Serialised: concurrent Logging reads are what earn a 429.
|
|
const sample = {
|
|
at,
|
|
cell: await readCellSample(deps.fetchImpl, request, dependencies.idToken, sleep),
|
|
director503: await readDirector503Minutes(reader, {
|
|
projectId: request.projectId,
|
|
directorService,
|
|
// Complete minutes only, held back past the publishing delay.
|
|
endedAt: minuteFloor(at - METRIC_DELAY_MS),
|
|
minutes: RESERVE_WATCH_RULES.director503.readMinutes
|
|
}),
|
|
ledger: await readLedgerSample(reader, { directorService, at }),
|
|
seatBehindRow: await readSeatBehindRowSample(reader, { directorService, cellId: request.cellId, at })
|
|
}
|
|
const result = evaluateWatchSample(state, sample, { unverifiableLimit: config.unverifiableLimit })
|
|
state = result.state
|
|
log(result.line)
|
|
await writeFile(config.stateFile, JSON.stringify(state))
|
|
if (state.state !== 'watching') break
|
|
await sleep(Math.max(0, at + SAMPLE_INTERVAL_MS - now()))
|
|
}
|
|
if (config.githubOutput) await writeOutcome(config.githubOutput, state)
|
|
return state
|
|
}
|
|
|
|
function writeOutcome(githubOutput, { state, verdict }) {
|
|
return appendFile(githubOutput, `state=${state}\nverdict=${(verdict ?? '').replace(/[\r\n]+/g, ' ')}\n`)
|
|
}
|
|
|
|
// The job's outcome. A watch that ended mid-segment (a crash, a timeout) records failed, which the
|
|
// workflow flips back in write mode: missing data is never healthy.
|
|
export async function recordOutcome(stateFile, githubOutput) {
|
|
const state = await loadState(stateFile).catch(() => null)
|
|
const outcome =
|
|
state && state.state !== 'watching' ? state : { state: 'failed', verdict: 'the watch stopped before a verdict' }
|
|
await writeOutcome(githubOutput, outcome)
|
|
return outcome
|
|
}
|
|
|
|
function breakGlassInputs(request, generation) {
|
|
return {
|
|
mode: 'write',
|
|
'cell-id': request.cellId,
|
|
set: 'admitMode=default',
|
|
'break-glass': `${BREAK_GLASS_CONFIRMATION} ${request.cellId}`,
|
|
'expected-generation': generation,
|
|
confirmation: `${WRITE_CONFIRMATION} ${request.cellId}`
|
|
}
|
|
}
|
|
|
|
async function readRuntimeRetrying(fetchImpl, request, idToken, sleep, log) {
|
|
for (let attempt = 1; ; attempt += 1) {
|
|
try {
|
|
return await readRuntime(fetchImpl, request, idToken, sleep)
|
|
} catch (error) {
|
|
if (attempt >= FLIP_BACK_RUNTIME_ATTEMPTS) throw error
|
|
log(JSON.stringify({ event: 'orca_relay_reserve_watch_runtime_retry', cellId: request.cellId, attempt, message: error instanceof Error ? error.message : String(error) }))
|
|
await sleep(FLIP_BACK_BACKOFF_MS * 2 ** (attempt - 1))
|
|
}
|
|
}
|
|
}
|
|
|
|
/**
|
|
* The flag tool's admitMode=db write at the generation read just now. A generation conflict (some
|
|
* other write landed first) is re-read and retried while the switch file still says reserve; a
|
|
* file that already says db is done. A cell that still does not answer after the runtime retries
|
|
* needs break glass, which only an operator may type.
|
|
*/
|
|
export async function flipBack(values, dependencies) {
|
|
const log = dependencies.log ?? console.log
|
|
const fetchImpl = dependencies.fetchImpl ?? fetch
|
|
const sleep = dependencies.sleep ?? ((ms) => new Promise((resolve) => setTimeout(resolve, ms)))
|
|
const base = dbFlipRequest(values)
|
|
const record = (result) =>
|
|
log(
|
|
JSON.stringify({
|
|
event: 'orca_relay_reserve_watch_flip_back',
|
|
mode: base.mode,
|
|
cellId: base.cellId,
|
|
written: result.written,
|
|
generation: result.generation ?? null,
|
|
...(result.alreadyDb ? { alreadyDb: true } : {})
|
|
})
|
|
)
|
|
for (let attempt = 1; ; attempt += 1) {
|
|
const current = await readCurrentObject(fetchImpl, base, dependencies.accessToken)
|
|
if ((current.object?.flags?.admitMode ?? 'db') !== 'reserve') {
|
|
const result = { written: false, alreadyDb: true, generation: current.generation }
|
|
record(result)
|
|
return result
|
|
}
|
|
try {
|
|
await readRuntimeRetrying(fetchImpl, base, dependencies.idToken, sleep, log)
|
|
} catch (error) {
|
|
const inputs = breakGlassInputs(base, current.generation)
|
|
log(JSON.stringify({ event: 'orca_relay_reserve_watch_break_glass_required', cellId: base.cellId, inputs }))
|
|
throw new Error(
|
|
`${base.cellId} does not answer runtime-status (${error instanceof Error ? error.message : error}), so ` +
|
|
'the flag tool cannot flip it back. Run Operate Relay Production Cell Flags with break glass: ' +
|
|
Object.entries(inputs).map(([key, value]) => `${key}=${value}`).join(', ')
|
|
)
|
|
}
|
|
try {
|
|
const result = await operateCellFlags(
|
|
{ ...base, expectedGeneration: current.generation },
|
|
{ ...dependencies, fetchImpl, log, sleep }
|
|
)
|
|
record(result)
|
|
return result
|
|
} catch (error) {
|
|
if (!error?.generationConflict || attempt >= FLIP_BACK_ATTEMPTS) throw error
|
|
log(JSON.stringify({ event: 'orca_relay_reserve_watch_flip_back_conflict', cellId: base.cellId, attempt, message: error.message }))
|
|
await sleep(FLIP_BACK_BACKOFF_MS * 2 ** (attempt - 1))
|
|
}
|
|
}
|
|
}
|
|
|
|
export const ALARM_EVENT = 'orca_relay_reserve_watch_alarm'
|
|
export const ALARM_LOG_NAME = 'orca-relay-reserve-watch'
|
|
export const ALARM_KINDS = ['refused', 'flip-back-failed']
|
|
|
|
// GitHub's workflow-command escaping, so a verdict cannot end the annotation or start another.
|
|
export const escapeWorkflowCommand = (text) => text.replace(/%/g, '%25').replace(/\r/g, '%0D').replace(/\n/g, '%0A')
|
|
const escapeWorkflowProperty = (text) => escapeWorkflowCommand(text).replace(/:/g, '%3A').replace(/,/g, '%2C')
|
|
|
|
/**
|
|
* Both kinds can leave the cell in reserve with nothing watching it. Only write mode pages: a
|
|
* dry-run watch never flips back, so its refusal or failure changes nothing for the cell.
|
|
*/
|
|
export async function raiseAlarm({ mode, cellId, projectId, kind, reason }, { runGcloud, log = console.log }) {
|
|
if (!ALARM_KINDS.includes(kind)) throw new Error(`--kind must be one of ${ALARM_KINDS.join(', ')}`)
|
|
const message =
|
|
kind === 'refused'
|
|
? `${cellId} may be in reserve mode and unwatched: the watch was refused (${reason}). Rerun it, or flip the cell to db with Operate Relay Production Cell Flags.`
|
|
: `${cellId} may still be in reserve mode with no watch: the flip back did not land (${reason}). Flip it to db by hand with Operate Relay Production Cell Flags (set=admitMode=db), or with the break-glass inputs logged above if the cell does not answer.`
|
|
log(`::error title=${escapeWorkflowProperty(`Reserve watch ${kind} for ${cellId}`)}::${escapeWorkflowCommand(message)}`)
|
|
if (mode !== 'write') return { paged: false }
|
|
const entry = JSON.stringify({ event: ALARM_EVENT, cellId, kind, reason })
|
|
try {
|
|
// The flag tool's project check: a missing --project-id must not reach gcloud as "undefined".
|
|
if (!/^[a-z][a-z0-9-]{4,62}$/.test(projectId ?? '')) throw new Error('--project-id is not a GCP project id')
|
|
await runGcloud(
|
|
['logging', 'write', ALARM_LOG_NAME, entry, '--payload-type=json', '--severity=ERROR', '--project', projectId],
|
|
{ timeoutMs: 30_000 }
|
|
)
|
|
} catch (error) {
|
|
log(`::error title=Reserve watch page failed::${escapeWorkflowCommand(`the alarm log line was not written, so nothing pages: ${error instanceof Error ? error.message : error}`)}`)
|
|
return { paged: false }
|
|
}
|
|
return { paged: true }
|
|
}
|
|
|
|
if (import.meta.url === pathToFileURL(process.argv[1] ?? '').href) {
|
|
const [command, ...rest] = process.argv.slice(2)
|
|
const { values } = parseArgs({
|
|
args: rest,
|
|
options: Object.fromEntries(
|
|
[
|
|
'mode',
|
|
'cell-id',
|
|
'cell-origin',
|
|
'project-id',
|
|
'director-origin',
|
|
'director-service',
|
|
'confirmation',
|
|
'duration-minutes',
|
|
'segment-minutes',
|
|
'unverifiable-limit',
|
|
'background-503-per-minute',
|
|
'flipped-at',
|
|
'state-file',
|
|
'github-output',
|
|
'verdict',
|
|
'kind'
|
|
].map((name) => [name, { type: 'string' }])
|
|
)
|
|
})
|
|
const idToken = process.env.ORCA_RELAY_ADMIN_ID_TOKEN
|
|
const runGcloud = (args, { timeoutMs }) =>
|
|
execFileAsync('gcloud', args, { maxBuffer: 64 * 1024 * 1024, timeout: timeoutMs, killSignal: 'SIGKILL' })
|
|
try {
|
|
if (command === 'alarm') {
|
|
await raiseAlarm(
|
|
{ mode: values.mode, cellId: values['cell-id'] ?? '', projectId: values['project-id'], kind: values.kind, reason: values.verdict ?? '' },
|
|
{ runGcloud }
|
|
)
|
|
// The alarm step is red, whatever the page did.
|
|
process.exitCode = 1
|
|
} else if (command === 'outcome') {
|
|
if (!values['state-file'] || !values['github-output']) throw new Error('--state-file and --github-output are required')
|
|
console.log(JSON.stringify({ event: 'orca_relay_reserve_watch_outcome', ...(await recordOutcome(values['state-file'], values['github-output'])) }))
|
|
} else if (!idToken) {
|
|
throw new Error('ORCA_RELAY_ADMIN_ID_TOKEN is required')
|
|
} else if (command === 'watch') {
|
|
const config = await parseWatchSegment(values)
|
|
const state = await watchSegment(config, {
|
|
idToken,
|
|
readRolloutLease: readProductionRolloutLease,
|
|
runGcloud
|
|
})
|
|
// A cell that was never reserve is the operator's mistake; one flipped back by hand is not.
|
|
if (state.state === 'refused') {
|
|
console.log(JSON.stringify({ event: 'orca_relay_reserve_watch_refused', verdict: state.verdict }))
|
|
process.exitCode = 1
|
|
}
|
|
} else if (command === 'flip-back') {
|
|
const accessToken = process.env.ORCA_RELAY_CONTROL_ACCESS_TOKEN
|
|
if (!accessToken) throw new Error('ORCA_RELAY_CONTROL_ACCESS_TOKEN is required')
|
|
if (!values.verdict) throw new Error('--verdict is required')
|
|
await flipBack(values, {
|
|
accessToken,
|
|
idToken,
|
|
audit: {
|
|
writtenBy: `reserve-watch/${process.env.GITHUB_TRIGGERING_ACTOR ?? process.env.GITHUB_ACTOR ?? 'unknown'}`,
|
|
runId: process.env.GITHUB_RUN_ID ?? 'local',
|
|
runAttempt: process.env.GITHUB_RUN_ATTEMPT ?? '0'
|
|
}
|
|
})
|
|
// A trip is a failed watch even once the cell is safely back on db.
|
|
console.log(JSON.stringify({ event: 'orca_relay_reserve_watch_verdict', mode: values.mode, verdict: values.verdict }))
|
|
process.exitCode = 1
|
|
} else {
|
|
throw new Error('command must be watch, outcome, flip-back or alarm')
|
|
}
|
|
} catch (error) {
|
|
console.log(
|
|
JSON.stringify({
|
|
event: 'orca_relay_reserve_watch_failed',
|
|
message: error instanceof Error ? error.message : String(error)
|
|
})
|
|
)
|
|
process.exitCode = 1
|
|
}
|
|
}
|