mirror of
https://github.com/stablyai/orca.git
synced 2026-11-04 11:07:42 +00:00
Jinwoo Hong
6e46d5ef33
* feat(relay): watch a reserve cell and flip it back on the first step 5 stop rule A manual, inert-until-dispatched workflow samples one reserve cell every minute for up to 300 minutes against the step 5 §9 stop rules, logs one JSON line per sample, and on the first trip (or N unreadable samples in a row) runs the flag tool's own admitMode=db write under the flag workflow's concurrency group. dry-run logs the flag tool's plan instead. Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * fix(relay): keep the reserve watch's flip back out of the flag group, and baseline 503s before the flip - The flip back holds no concurrency group: GitHub keeps one pending run per group and a newer one cancels it. The flag tool's CAS orders it instead; a generation conflict is re-read and retried (5 tries, backoff) while the switch file still says reserve. - The 503 background comes from the 10 minutes before a required flipped-at, or an explicit value; neither is refused. - A failed or cancelled write-mode watch flips back too. An unreachable cell fails with the exact break-glass inputs. - Missing 503 data (empty series, unpublished minute, unparseable time) is unverifiable. - Seat-behind-row key adds the director instance and cell; six segments share one command. Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * fix(relay): read the reserve watch's 503 publishing front from every response code A quiet fleet has no 503 series, so the watch read it as unverifiable and flipped a healthy cell back within minutes. The 503 read now groups request_count by response code in one query; the newest minute any code has a point for is how far the series is published, so a 503 minute at or before it with no point is zero and minutes past it are left out. Also: the flip-back job's steps run on always(), flipped-at ages against the watch's start in later segments, a pending pre-flip background no longer adds to the unverifiable streak, the flip back retries runtime-status before calling for break glass, and the concurrency rationale lives once in the docs. Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * docs(relay): state what a cancel during the flip-back lease wait can leave, and the 503 publish-skew assumption Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * fix(relay): fail a segment closed on a corrupt watch state file, and keep no checkout token on disk Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * fix(relay): alarm on a failed flip back, refuse a watch under a rollout lease, wait for reserve to apply, and control the duplicate read - A flip back whose write or plan never finished (no ID token, a failed login, a silent cell) raises an ::error:: annotation and a FLIP BACK FAILED summary, as the clock-skew monitor does. - The first segment reads the Cloud SQL rollout lease through the lease action's own reader and refuses to start while any run holds it, or when it cannot be read. - A cell not yet reporting reserve is waited on for up to 10 minutes, judging nothing, before the watch ends refused without a write. - The duplicate-seat read takes the per-minute ledger summary in the same query and window; with no fresh summary in it the sample is unverifiable, not zero duplicates. Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * fix(relay): page on a refused reserve watch or a failed flip back A refused watch can leave a cell in reserve and unwatched, the same as a flip back that did not land. Both now raise an escaped ::error:: annotation through one `alarm` command, and in write mode write orca_relay_reserve_watch_alarm with gcloud logging write. A new log-based metric and alert policy page on any such line, per cell, through relay_alert_notification_channels, and the deploy identity gets roles/logging.logWriter in production to write it. Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * fix(relay): name the log-writer risk, check the alarm's project, and say a failed login cannot page Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * docs(relay): note a force-cancel can skip the reserve-watch flip-back write
110 lines
6.6 KiB
JavaScript
110 lines
6.6 KiB
JavaScript
import assert from 'node:assert/strict'
|
|
import { readFileSync } from 'node:fs'
|
|
import test from 'node:test'
|
|
import { fileURLToPath } from 'node:url'
|
|
|
|
// Why: each metric keys on the event name and fields the relay writes. A rename on either side
|
|
// silences the alert without failing anything.
|
|
|
|
const read = (relative) => readFileSync(fileURLToPath(new URL(relative, import.meta.url)), 'utf8')
|
|
const terraform = read('../../infra/terraform/relay-observability.tf')
|
|
const deadMan = read('../../apps/relay/src/cell-reserve-dead-man.ts')
|
|
const registry = read('../../apps/relay/src/host-session-registry.ts')
|
|
const server = read('../../apps/relay/src/relay-server.ts')
|
|
const poller = read('../../apps/relay/src/shadow-seat-directory.ts')
|
|
const watch = read('./watch-relay-reserve-cell.mjs')
|
|
const githubActions = read('../../infra/terraform/relay-github-actions.tf')
|
|
|
|
const block = (kind, name) => {
|
|
const body = new RegExp(`resource "${kind}" "${name}" \\{([\\s\\S]*?)\\n\\}`).exec(terraform)?.[1]
|
|
assert.ok(body, `${kind}.${name} not found in relay-observability.tf`)
|
|
return body
|
|
}
|
|
|
|
test('the metric counts the event and cell the dead-man writes', () => {
|
|
assert.match(deadMan, /event: 'orca_relay_cell_reserve_dead_man_tripped',\s*cellId: this\.cellId,/)
|
|
const metric = block('google_logging_metric', 'relay_cell_reserve_dead_man_tripped')
|
|
assert.ok(metric.includes('jsonPayload.event=\\"orca_relay_cell_reserve_dead_man_tripped\\"'))
|
|
assert.ok(metric.includes('cell_id = "EXTRACT(jsonPayload.cellId)"'))
|
|
})
|
|
|
|
test('the alert fires on one trip per cell and notifies', () => {
|
|
const alert = block('google_monitoring_alert_policy', 'relay_cell_reserve_dead_man_tripped')
|
|
// One line per switch-file generation, so a higher bar misses a trip.
|
|
assert.match(alert, /threshold_value = 0\n/)
|
|
assert.match(alert, /notification_channels = var\.relay_alert_notification_channels/)
|
|
assert.ok(alert.includes('metric.type=\\"logging.googleapis.com/user/orca_relay_cell_reserve_dead_man_tripped\\"'))
|
|
assert.ok(alert.includes('group_by_fields = ["metric.label.\\"cell_id\\""]'))
|
|
})
|
|
|
|
const notifyingOnAnyLine = (name, event, resource, group) => {
|
|
const alert = block('google_monitoring_alert_policy', name)
|
|
assert.match(alert, /threshold_value = 0\n/)
|
|
assert.match(alert, /notification_channels = var\.relay_alert_notification_channels/)
|
|
assert.ok(alert.includes(`resource.type=\\"${resource}\\" AND metric.type=\\"logging.googleapis.com/user/${event}\\"`))
|
|
assert.ok(alert.includes(`group_by_fields = ["metric.label.\\"${group}\\""]`))
|
|
}
|
|
|
|
test('the flip-back alerts count the lines the cell writes, per cell', () => {
|
|
for (const event of ['orca_relay_cell_reregistration_gave_up', 'orca_relay_cell_reregistration_stalled']) {
|
|
// logIdentity() carries cellId; both lines name the reason.
|
|
assert.match(registry, new RegExp(`event: '${event}',\\s*\\.\\.\\.this\\.logIdentity\\(\\),[\\s\\S]{0,200}reason:`))
|
|
const name = event.replace('orca_', '')
|
|
const metric = block('google_logging_metric', name)
|
|
assert.ok(metric.includes(`resource.type=\\"gce_instance\\" AND jsonPayload.event=\\"${event}\\"`))
|
|
assert.ok(metric.includes('cell_id = "EXTRACT(jsonPayload.cellId)"'))
|
|
assert.ok(metric.includes('reason = "EXTRACT(jsonPayload.reason)"'))
|
|
notifyingOnAnyLine(name, event, 'gce_instance', 'cell_id')
|
|
}
|
|
assert.match(registry, /cellId: this\.config\.cellId/)
|
|
})
|
|
|
|
test('the director alert counts its placement-off line on the director service', () => {
|
|
const event = 'orca_relay_reserve_placement_off_with_reserve_cells'
|
|
assert.match(server, new RegExp(`event: '${event}',\\s*cells,`))
|
|
const metric = block('google_logging_metric', 'relay_reserve_placement_off_with_reserve_cells')
|
|
assert.ok(
|
|
metric.includes(
|
|
`resource.type=\\"cloud_run_revision\\" AND resource.labels.service_name=\\"\${var.relay_cloud_run_service_name}\\" AND jsonPayload.event=\\"${event}\\"`
|
|
)
|
|
)
|
|
notifyingOnAnyLine('relay_reserve_placement_off_with_reserve_cells', event, 'cloud_run_revision', 'revision')
|
|
})
|
|
|
|
test('the withheld dead-cell verdict alert counts every director withheld line, gated off by default', () => {
|
|
const event = 'orca_relay_dead_cell_verdict_withheld'
|
|
assert.match(poller, new RegExp(`event: '${event}', cellId, \\.\\.\\.verdict`))
|
|
const metric = block('google_logging_metric', 'relay_dead_cell_verdict_withheld')
|
|
assert.ok(metric.includes(`jsonPayload.event=\\"${event}\\""`))
|
|
assert.ok(metric.includes('withheld = "EXTRACT(jsonPayload.withheld)"'))
|
|
const alert = block('google_monitoring_alert_policy', 'relay_dead_cell_verdict_withheld')
|
|
for (const body of [metric, alert]) assert.match(body, /count = var\.relay_dead_cell_verdict_alert \? 1 : 0/)
|
|
assert.match(alert, /threshold_value = 0\n/)
|
|
assert.match(alert, /notification_channels = var\.relay_alert_notification_channels/)
|
|
assert.ok(alert.includes('logging.googleapis.com/user/${google_logging_metric.relay_dead_cell_verdict_withheld[0].name}'))
|
|
assert.ok(alert.includes('group_by_fields = ["metric.label.\\"withheld\\""]'))
|
|
assert.match(read('../../infra/terraform/variables.tf'), /variable "relay_dead_cell_verdict_alert" \{[\s\S]*?default = false/)
|
|
assert.match(read('../../infra/terraform/environments/staging.tfvars'), /relay_dead_cell_verdict_alert\s+= true/)
|
|
})
|
|
|
|
test('the reserve watch alarm pages on any line the watch runner writes, per cell', () => {
|
|
// The runner writes a global-resource entry under its own log name.
|
|
assert.match(watch, /export const ALARM_EVENT = 'orca_relay_reserve_watch_alarm'/)
|
|
assert.match(watch, /export const ALARM_LOG_NAME = 'orca-relay-reserve-watch'/)
|
|
assert.match(watch, /JSON\.stringify\(\{ event: ALARM_EVENT, cellId, kind, reason \}\)/)
|
|
const metric = block('google_logging_metric', 'relay_reserve_watch_alarm')
|
|
assert.ok(
|
|
metric.includes(
|
|
'resource.type=\\"global\\" AND logName=\\"projects/${var.project_id}/logs/orca-relay-reserve-watch\\" AND jsonPayload.event=\\"orca_relay_reserve_watch_alarm\\"'
|
|
)
|
|
)
|
|
assert.ok(metric.includes('cell_id = "EXTRACT(jsonPayload.cellId)"'))
|
|
notifyingOnAnyLine('relay_reserve_watch_alarm', 'orca_relay_reserve_watch_alarm', 'global', 'cell_id')
|
|
// The deploy identity the workflow runs as can write that line, and only in production.
|
|
const grant = /resource "google_project_iam_member" "github_log_writer" \{([\s\S]*?)\n\}/.exec(githubActions)?.[1]
|
|
assert.ok(grant, 'github_log_writer grant not found')
|
|
assert.match(grant, /count = local\.relay_create_production_ops_identity \? 1 : 0/)
|
|
assert.match(grant, /role = "roles\/logging\.logWriter"/)
|
|
assert.match(grant, /member = google_service_account\.github_deploy\[0\]\.member/)
|
|
})
|