mirror of
https://github.com/stablyai/orca.git
synced 2026-09-22 00:02:31 +00:00
Jinwoo Hong
ce5d8c02d4
* fix(relay): wait out a cold proxy at boot instead of exiting the cell A cell container starts its relay process beside a cloud-sql-proxy that is itself still dialling. The first pool acquire therefore competes with a proxy cold start, and the 2s connect timeout that protects the request path fires before the proxy is listening. `openRelayDatabase` rejects out of the region backfill, the top-level await rejects, and the process exits; COS restarts the container and the next boot succeeds 1-3s later. The 2026-09-18 fleet roll saw 0-7 of these per cell, including on cells with zero hosts, so it is a property of the boot sequence rather than of database load. The boot open now retries on transient errors only, inside a 45s wall-clock window with exponential backoff from 250ms to 4s. The classifier is the one the request path already uses, so a rejected credential or a bad URL still exits on the first attempt. Each wait logs `orca_relay_boot_database_retry` and a give-up logs `orca_relay_boot_database_failed`, both with the bounded error category, so a rollout can tell a slow boot from a stuck one without reading container exit codes. The bounded startup retry is lifted out of `reconcileCellAdmissionAtStartup`, which had the same loop; its attempt budget, flat delay, and both log events are unchanged (a flat delay is a cap equal to the base). * fix(relay): retry the boot open only when Postgres is unreachable The boot open re-runs the schema apply, and applyPostgresSchema refuses to repeat a DDL lock timeout on purpose: relation locks are granted in queue order, so a repeat parks every writer behind the same statement again. Gating the boot retry on the full request-path classifier would have re-queued it up to 16 times in 45s on sustained 55P03 - the mechanism behind the 2026-09-16 outage. The boot call site now has its own predicate: pool connect failures (both connect-timeout messages and an acquire-marked early-ended socket) plus 08001 and 08006. Lock and overload SQLSTATEs - 55P03, 57014, 53300 - exit on the first attempt. The retry predicate moves onto the policy because what a step re-runs, not the request path, decides what it may repeat; the startup reconcile keeps the full classifier, which is what lets it wait out 55P03.
72 lines
2.5 KiB
TypeScript
72 lines
2.5 KiB
TypeScript
import { openRelayDatabase, type RelayDatabase, type RelayDatabaseOpenInput } from './database.js'
|
|
import { retryTransientDatabaseStartup } from './database-startup-retry.js'
|
|
import {
|
|
isPostgresPoolConnectFailure,
|
|
isPostgresPoolConnectTimeout
|
|
} from './postgres-pool-pressure.js'
|
|
import { postgresErrorCodeCategory } from './postgres-query-failure.js'
|
|
|
|
// Only a failure to reach Postgres at all. A retry here re-runs the schema
|
|
// apply, and applyPostgresSchema refuses to repeat a DDL lock timeout on
|
|
// purpose: relation locks are granted in queue order, so a repeat parks every
|
|
// writer behind the same statement again. 55P03, 57014 and 53300 therefore stay
|
|
// terminal at boot even though the request path calls them transient.
|
|
function isBootDatabaseUnreachable(error: unknown): boolean {
|
|
const code = postgresErrorCodeCategory(error)
|
|
return isPostgresPoolConnectFailure(error) || code === '08001' || code === '08006'
|
|
}
|
|
|
|
// A cell boots beside a cloud-sql-proxy that is itself still dialling, so the
|
|
// first pool acquire can outrun the 2s connect timeout that protects the
|
|
// request path. The window is longer than a proxy cold start and shorter than
|
|
// the restart loop it replaces.
|
|
const BOOT_OPEN_RETRY = {
|
|
attempts: 20,
|
|
windowMs: 45_000,
|
|
baseDelayMs: 250,
|
|
maxDelayMs: 4_000,
|
|
jitterMs: 250,
|
|
isRetryable: isBootDatabaseUnreachable
|
|
}
|
|
|
|
function bootDatabaseErrorFields(error: unknown): Record<string, unknown> {
|
|
return {
|
|
code: postgresErrorCodeCategory(error),
|
|
connectionTimeout: isPostgresPoolConnectTimeout(error)
|
|
}
|
|
}
|
|
|
|
export async function openRelayDatabaseAtBoot(
|
|
input: RelayDatabaseOpenInput,
|
|
open: (input: RelayDatabaseOpenInput) => Promise<RelayDatabase> = openRelayDatabase
|
|
): Promise<RelayDatabase> {
|
|
return await retryTransientDatabaseStartup(
|
|
async () => await open(input),
|
|
BOOT_OPEN_RETRY,
|
|
{
|
|
onRetry: ({ attempt, delayMs, error }) =>
|
|
console.warn(
|
|
JSON.stringify({
|
|
event: 'orca_relay_boot_database_retry',
|
|
attempt,
|
|
delayMs,
|
|
...bootDatabaseErrorFields(error)
|
|
})
|
|
),
|
|
onRecovered: ({ attempts }) =>
|
|
console.warn(
|
|
JSON.stringify({ event: 'orca_relay_boot_database_recovered', attempts })
|
|
),
|
|
onGaveUp: ({ attempts, error, retryable }) =>
|
|
console.warn(
|
|
JSON.stringify({
|
|
event: 'orca_relay_boot_database_failed',
|
|
attempts,
|
|
retryable,
|
|
...bootDatabaseErrorFields(error)
|
|
})
|
|
)
|
|
}
|
|
)
|
|
}
|