mirror of
https://github.com/stablyai/orca.git
synced 2026-11-03 10:37:35 +00:00
fceb475e44944dd2e1908d08d912942caf917dcf
32
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Jinwoo Hong
|
2a15a50691 |
fix(relay): retire a pooled Postgres connection that turned slow or lossy (#33900)
Director only. When one pooled connection is far slower than its peers (median over 25 ms and 3x theirs, or third-slowest statement in 60 s over 200 ms and 5x every peer's slowest), it is destroyed on release and the pool dials a fresh one. At most one per pool per 10 s; never mid-transaction; no change in connection count. Fixes the 10-26 director stalls where one connection stalled the sticky lane. |
||
  OrcaWinandOrcaWin
|
7d6603aa17 |
[comment audit] Complete repository-wide comment cleanup (#33249)
* [comment audit] Align legacy failure-read counts with cached display * [comment audit] Condense shared provider lifecycle and browser recovery comments [skip ci] * [comment audit] Condense shared generation, path and preview comments [skip ci] * [comment audit] Condense host, file and Git contract rationale [skip ci] * [comment audit] Tighten Git lifecycle and provider contract comments [skip ci] * [comment audit] Clarify shared host authority and launch isolation [skip ci] * [comment audit] simplify local access mobile and native chat contracts [skip ci] * [comment audit] simplify native chat presentation and state contracts [skip ci] * [comment audit] simplify native runtime and provider contracts [skip ci] * [comment audit] simplify migration and orchestration contracts [skip ci] * [comment audit] simplify plugin trust and process capture rationale [skip ci] * [comment audit] simplify terminal delivery and remote compatibility rationale [skip ci] * [comment audit] clarify remote recovery and launch wire rationale [skip ci] * [comment audit] shorten shared RPC compatibility and runtime evidence comments [skip ci] * [comment audit] simplify secure storage shell startup and sidebar rationale [skip ci] * [comment audit] clarify sidebar, skill and SSH invariants [skip ci] * [comment audit] condense structured chat delivery and timing rationale [skip ci] * [comment audit] condense terminal replay and telemetry invariants [skip ci] * [comment audit] clarify launch permission and terminal delivery rationale [skip ci] * [comment audit] clarify workspace ownership and layout contracts [skip ci] * [comment audit] wait for fitted and painted SSH terminal links * [comment audit] clarify layout persistence and session ownership [skip ci] * [comment audit] clarify host routing and workspace lifecycle contracts [skip ci] * [comment audit] clarify startup persistence and app lifetime comments [skip ci] * [comment audit] clarify recovery and activity boundaries [skip ci] * [comment audit] condense activity and artifact invariants [skip ci] * [comment audit] condense automation authority contracts [skip ci] * [comment audit] simplify automation scheduling and history contracts [skip ci] * [comment audit] clarify browser chrome and annotation lifecycles [skip ci] * [comment audit] clarify browser routing and retained guest ownership [skip ci] * [comment audit] clarify browser recovery and document ownership comments [skip ci] * [comment audit] restore hosted runners required by native IME fixtures * [comment audit] shorten preview, palette and dashboard invariants [skip ci] * [comment audit] clarify terminal preview, dashboard and retention comments [skip ci] * [comment audit] clarify diff-note recovery and editor lifecycle comments [skip ci] * [comment audit] shorten rich-text and combined-diff ownership comments [skip ci] * [comment audit] shorten autosave, draft recovery and model ownership comments [skip ci] * [comment audit] shorten save-queue, notebook and Markdown constraints [skip ci] * [comment audit] shorten Monaco, PDF and rich Markdown safeguards [skip ci] * [comment audit] clarify rich Markdown source and selection safeguards [skip ci] * [comment audit] clarify editor loading and emulator stream lifetimes [skip ci] * [comment audit] trim emulator and feature tour narration [skip ci] * [comment audit] clarify floating workspace and GitHub cache ownership [skip ci] * [comment audit] trim provider controls and Linear filter explanations [skip ci] * [comment audit] clarify pairing and chat resume lifetimes [skip ci] * [comment audit] clarify chat delivery and transcript lifetimes [skip ci] * [comment audit] clarify session options and host-owned status [skip ci] * [comment audit] clarify draft durability and live-status precedence [skip ci] * [comment audit] Trim chat send and model ownership explanations [skip ci] * [comment audit] Shorten transcript ownership and send recovery comments [skip ci] * [comment audit] Simplify native chat composer and transcript hook comments [skip ci] * [comment audit] Shorten native chat scroll and structured session explanations [skip ci] * [comment audit] Trim workspace picker and onboarding commentary [skip ci] * [comment audit] Simplify pet rendering, review cache and quick open comments [skip ci] * [comment audit] Trim sidebar and session history explanations [skip ci] * [comment audit] Condense session resume and review checks comments [skip ci] * [comment audit] Shorten checks state and file explorer refresh comments [skip ci] * [comment audit] Trim plugin bridge and source-control action rationale [skip ci] * [comment audit] Trim source-control listing and review rationale [skip ci] * [comment audit] Shorten sidebar lifecycle and explorer safeguards [skip ci] * [comment audit] Condense Settings account and appearance rationale [skip ci] * [comment audit] Trim pairing and repository Settings explanations [skip ci] * [comment audit] Condense Settings search SSH and terminal safeguards [skip ci] * [comment audit] Shorten Settings voice and provider ownership comments [skip ci] * [comment audit] Condense Settings draft persistence and host-state rationale [skip ci] * [comment audit] Trim Settings hooks and sidebar setup rationale [skip ci] * [comment audit] Condense sidebar deletion and board-filter safeguards [skip ci] * [comment audit] Condense worktree cards, host ownership and Markdown contracts [skip ci] * [comment audit] Condense sidebar drag, folder and filtering contracts [skip ci] * [comment audit] Condense sidebar parity, sleep and host ownership contracts [skip ci] * [comment audit] Prune focus-read baseline after comment cleanup * [comment audit] Clarify sidebar visibility, board drops and agent evidence [skip ci] * [comment audit] Tighten sidebar listing, drag and reveal contracts [skip ci] * [comment audit] Clarify viewport geometry and skill update safeguards [skip ci] * [comment audit] Tighten skill inventory, usage and host status comments [skip ci] * [comment audit] Clarify usage ownership and resource kill safeguards [skip ci] * [comment audit] Tighten tab selection, rename and search contracts [skip ci] * [comment audit] Tighten tab-strip geometry and drag ownership comments [skip ci] * [comment audit] Tighten Tasks recovery, pagination and mutation comments [skip ci] * [comment audit] Tighten provider loading and terminal recovery comments [skip ci] * [comment audit] Tighten completion, cadence and PTY ownership comments [skip ci] * [comment audit] Clarify terminal restore and parked-pane boundaries [skip ci] * [comment audit] Tighten PTY startup and replay explanations [skip ci] * [comment audit] Clarify PTY identity and delivery ownership [skip ci] * [comment audit] Shorten remote reconnect and input replay rationale [skip ci] * [comment audit] Condense terminal parking clipboard and file link comments [skip ci] * [comment audit] Clarify IME ownership and trim trace history [skip ci] * [comment audit] Shorten pane recovery and parked watcher explanations [skip ci] * [comment audit] Condense snapshot replay and visibility recovery rationale [skip ci] * [comment audit] Shorten terminal parking and keyboard ownership rationale [skip ci] * [comment audit] Trim renderer lifecycle and menu rationale [skip ci] * [comment audit] Condense workspace cleanup ownership and settlement notes [skip ci] * [comment audit] Clarify renderer hook evidence and source ownership [skip ci] * [comment audit] Tighten IPC ownership and reconnect explanations [skip ci] * [comment audit] Trim renderer focus, paste and attention comments [skip ci] * [comment audit] Shorten scroll, locale and agent lifecycle comments [skip ci] * [comment audit] Trim agent launch, resume and ownership commentary [skip ci] * [comment audit] Shorten account, browser and editor comments [skip ci] * [comment audit] Clarify host-mirror waits, previews and input ownership [skip ci] * [comment audit] Condense launch recovery and editor tokenization comments [skip ci] * [comment audit] Condense workspace selection and terminal pane rationale [skip ci] * [comment audit] Condense terminal fitting and rendering recovery comments [skip ci] * [comment audit] Condense terminal input and host routing explanations [skip ci] * [comment audit] Condense ownership and sleeping-session recovery rationale [skip ci] * [comment audit] Condense structured launch and terminal link explanations [skip ci] * [comment audit] Condense terminal ownership and workspace file routing rationale [skip ci] * [comment audit] Condense session persistence and workspace activation rationale [skip ci] * [comment audit] Condense workspace routing and browser ownership rationale [skip ci] * [comment audit] Condense mirror recovery and layout authority rationale [skip ci] * [comment audit] Condense runtime transport and file safety rationale [skip ci] * [comment audit] Condense runtime graph and structured session rationale [skip ci] * [comment audit] Condense browser creation and remote tab recovery rationale [skip ci] * [comment audit] Condense snapshot authority and publisher recovery rationale [skip ci] * [comment audit] Condense frame, startup and provider cache rationale [skip ci] * [comment audit] Condense host twin and agent lifecycle rationale [skip ci] * [comment audit] Condense browser authority and editor cleanup rationale [skip ci] * [comment audit] Condense editor buffer identity and review cache rationale [skip ci] * [comment audit] Condense provider lifetime and repository ownership rationale [skip ci] * [comment audit] Condense reconnect, hydration and terminal retirement rationale [skip ci] * [comment audit] Condense recovery, reconciliation and persistence rationale [skip ci] * [comment audit] Condense cleanup ownership and listing admission rationale [skip ci] * [comment audit] Condense teardown ordering and terminal ownership rationale [skip ci] * [comment audit] Condense recovery ledger and web host-boundary rationale [skip ci] * [comment audit] Condense web reconnect and credential lifecycle rationale [skip ci] * [comment audit] Condense cloud rollout and assignment safety rationale [skip ci] * [comment audit] Condense relay session and database recovery rationale [skip ci] * [comment audit] Condense relay placement and protocol compatibility rationale [skip ci] * [comment audit] Condense build and packaging constraints [skip ci] * [comment audit] Condense CI gates and validation isolation rationale [skip ci] * [comment audit] Condense catalog identity and browser validation rationale [skip ci] * [comment audit] Condense mobile browser and frame-budget rationale [skip ci] * [comment audit] Shorten mobile route and browser test explanations [skip ci] * [comment audit] Condense editor and session seam test rationale [skip ci] * [comment audit] Trim session and terminal closure history [skip ci] * [comment audit] Shorten terminal browser and census explanations [skip ci] * [comment audit] Preserve root-only E2E diagnostic baseline * [comment audit] Condense mobile seam and HTML preview test rationale [skip ci] * [comment audit] Condense preview instrumentation and render harness rationale [skip ci] * [comment audit] Condense bundle contracts and native-build constraints [skip ci] * [comment audit] Shorten native packaging and CI routing rationale [skip ci] * [comment audit] Condense release tooling and benchmark constraints [skip ci] * [comment audit] Condense packaging and platform gate rationale [skip ci] * [comment audit] Shorten mobile route and history rationale [skip ci] * [comment audit] Condense mobile browser and component rationale [skip ci] * [comment audit] Shorten drawer and workspace-form rationale [skip ci] * [comment audit] Trim mobile review and editor-document explanations [skip ci] * [comment audit] Condense editor lifecycle and Markdown explanations [skip ci] * [comment audit] Trim mobile file, diagnostics and dictation commentary [skip ci] * [comment audit] Simplify mobile host-list and workspace-view explanations [skip ci] * [comment audit] Clarify mobile shell lifetime and bridge delivery constraints [skip ci] * [comment audit] Condense bridge stream, native request and audio rationale [skip ci] * [comment audit] Reduce bridge schema and page-client commentary [skip ci] * [comment audit] Tighten mobile page bridge contracts [skip ci] * [comment audit] Clarify mobile cache and session constraints [skip ci] * [comment audit] Condense mobile route and lifecycle rationale [skip ci] * [comment audit] Tighten mobile shell hook timing rationale [skip ci] * [comment audit] Clarify route handoff and notification compatibility [skip ci] * [comment audit] Clarify push registration and dictation cleanup constraints [skip ci] * [comment audit] Condense keyboard geometry and native media ownership rationale [skip ci] * [comment audit] Tighten mobile chat scroll and composer concurrency rationale [skip ci] * [comment audit] Condense mobile reply compatibility and upload ownership rationale [skip ci] * [comment audit] Condense optimistic echo and scroll ownership rationale [skip ci] * [comment audit] Tighten stream replay and queued session ownership rationale [skip ci] * [comment audit] Condense mobile composer and reply-schema constraints [skip ci] * [comment audit] Clarify mobile transcript recovery and session ownership [skip ci] * [comment audit] Clarify mobile queue and source-control compatibility [skip ci] * [comment audit] Condense storage and task launch constraints [skip ci] * [comment audit] Clarify task schemas and terminal document ownership [skip ci] * [comment audit] Tighten terminal input and viewport rationale [skip ci] * [comment audit] Shorten terminal and RPC recorder contracts [skip ci] * [comment audit] Condense recorder engine and connection contracts [skip ci] * [comment audit] Shorten host credential and relay recovery comments [skip ci] * [comment audit] Tighten pairing recovery and bundle transport rationale [skip ci] * [comment audit] Clarify RPC delivery and workspace catalog boundaries [skip ci] * [comment audit] Condense catalog and relay entry contracts [skip ci] * [comment audit] Shorten shutdown and SSH relay delivery contracts [skip ci] * [comment audit] Condense WSL relay and E2E regression rationale [skip ci] * [comment audit] Shorten cross-version harness and renderer fixture explanations [skip ci] * [comment audit] Condense paired-host and E2E lifecycle rationale [skip ci] * [comment audit] Shorten layout oracle and regression explanations [skip ci] * [comment audit] Condense paired recovery and ownership regression contracts [skip ci] * [comment audit] Shorten restart, sidebar and SSH lifecycle contracts [skip ci] * [comment audit] Condense terminal rendering and IME regression rationale [skip ci] * [comment audit] Shorten terminal recovery and benchmark contracts [skip ci] * [comment audit] Condense tooling and Windows lifecycle proof rationale [skip ci] * [comment audit] Condense workflow routing and release rationale [skip ci] * [comment audit] Condense mobile security and installer contracts [skip ci] * [comment audit] Shorten native rationale and stylesheet workarounds [skip ci] * [comment audit] Shorten Windows visibility probe rationale [skip ci] * [comment audit] Condense embedded release and signing rationale [skip ci] * [comment audit] Retain current main test behavior in final reconciliation [skip ci] * [comment audit] Complete final comment review across current source inventory * [comment audit] Handle consolidated audit reports before final CI * [comment audit] Preserve credential storage census detail [skip ci] * [comment audit] Scale broad changed E2E selections with verified timing shards * [comment audit] Align E2E assertions with browser close and settled session metadata * [comment audit] Match existing E2Es to current resource and windowing contracts * [comment audit] Observe the shared server outage notice in reconnect coverage * [comment audit] Make existing renderer probes measure settled tab and PTY work * [comment audit] Align palette E2E selector with task search * [comment audit] Refresh remaining palette search selectors * [comment audit] Keep selector correction within selected audit specs * [comment audit] Await emoji fixture settings inside renderer evaluation * [comment audit] Cancel fixture fit retries between terminal tests --------- Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com> |
||
|
Jinwoo Hong
|
d96dca39da |
fix(relay): place against min(configured, cell-reported) connection cap (#33039)
The director now accepts a cell heartbeat whose reported connection cap is at or below its configured cap, and places with the lower of the two; a reported cap above the configured one is still refused. A cap-only change keeps the cell's stored capacity rows. This lets the US cells' configured cap be raised once, ahead of their rolls, so an old process at the lower cap stays live until it is replaced, instead of desktops pointed at it being refused during a resize. |
||
|
Jinwoo Hong
|
8f117e19b1 |
fix(relay): destroy a pooled client whose session the server ended (#29573)
* fix(relay): destroy a pooled client whose session the server ended, and mark work that never ran pg keeps calling a client queryable until its socket's end arrives, so a client released after 57P01 (pg_terminate_backend, shutdown) or a dropped socket went back to the pool and failed the next caller's statement with 'Connection terminated unexpectedly'. Such clients are now released with the error, so pg-pool discards them. Also marks failures where no statement of the operation reached the server (acquire, a failed BEGIN, a client dead at checkout) so a caller can tell work safe to re-run. * fix(relay): skip ROLLBACK on an ended session, drop unused nothing-ran marking Adds 57P04/57P05 to the session-ended codes and covers 08xxx, a dropped socket, an ordinary 23505, and a real pg_terminate_backend. Claude-Session: f2fe8cb5 |
||
|
Jinwoo Hong
|
88accb2e4a |
feat(relay): log where the sticky lane's slot time goes, per pooled connection (#28908)
* feat(relay): log what the sticky slot spends its hold on, per pooled connection * fix(relay): time every sticky round trip at the PostgreSQL client, pool wait apart * docs(relay): say which round trips the sticky slot tally covers |
||
|
Jinwoo Hong
|
a18883130f |
fix(relay): database sweeps and ops reads skip reserve-mode cells (step-5 census, inert) (#27046)
Rebased onto main as one commit after its parent squash-merged. |
||
|
Jinwoo Hong
|
9c41d9e8d8 |
feat(relay): per-cell switches for the rejection fence and the lost-reply margin (#27057)
* feat(relay): per-cell switches for the rejection fence and the lost-reply margin, and report ignored keys rejectionFence (default on) turns the unhandled-rejection fence off without a roll. readTimeoutMarginMs (1-60 s) replaces the boot margin per query through pg's query_timeout. The applied log line and flagsApplied name keys this image does not know. * feat(relay): runtime-status names the switches and values this image accepts The flag tool checks a write against it, so it can refuse ticketCheck=enforce (or any key) on an image that would void or drop it. |
||
|
Jinwoo Hong
|
e72044a936 |
fix(relay): a lost database reply ends its connection within statement timeout + 10 s (#26829)
* fix(relay): a lost database reply ends its connection within statement timeout + 2 s The request pool sets query_timeout to the statement timeout plus 2 s. A read timeout poisons its transaction: later statements fail at once, COMMIT never runs (even when the operation caught the error), no ROLLBACK queues behind the dead statement, and the client is released with the error so pg-pool destroys it. Autocommit statements destroy their client too. The server bounds an orphaned idle backend with idle_session_timeout 30 s. A read timeout is answered 503 and never retried, and a renewal batch that loses its reply fails whole instead of fanning out. * fix(relay): read timeout margin 10 s, configurable, so routine 5-7 s database stalls never trip it A whole-VM stall delays replies the server already timed. With a 2 s margin the read timeout fired inside routine stalls, destroying healthy clients and failing requests that succeed today. The margin is now 10 s (ORCA_RELAY_POSTGRES_READ_TIMEOUT_MARGIN_MS), and a real-server test holds a reply 6.5 s and requires it delivered on the same connection. The fix level rises to 2 here, with the lost-reply fix. * docs(relay): name the startup retry that can see a read timeout, and why it is safe Boot open retries only an unreachable database. The director's startup cell reconcile does retry a lost reply, which is safe: it re-runs upserts and limit-change deletes that converge on the same rows. * fix(relay): answer 503, not 500, when Postgres ends the session mid-request Staging B1: pg_terminate_backend during /v1/assign returned 500 29 times. 57P01/57P02 and 08000/08003 join the transient SQLSTATEs; pg's 'Connection terminated' and ECONNRESET count only when the database layer threw them. transaction() still never retries them. * fix(relay): answer 503 when the pool acquire fails on a dead socket Staging B1 re-test: a Cloud SQL restart failed PostgresDatabase.transaction's acquire with 'write EPIPE', answered 500. An acquire failure with EPIPE, ECONNRESET, ECONNREFUSED, ETIMEDOUT or EHOSTUNREACH is now a pool connect failure (no statement ran), and acquire failures count as database-layer errors. EPIPE mid-statement is treated like ECONNRESET. |
||
|
Jinwoo Hong
|
dc69efa113 |
fix(relay): a database fault reaching unhandledRejection no longer kills the process (#26817)
* fix(relay): a database fault reaching unhandledRejection no longer kills the process
A rejection is fenced only when the relay's database layer threw or
rethrew it and it is a pool acquire failure, pg's read timeout, a dropped
connection, or a server error in classes 08/23/40/53/55/57/58. Everything
else stays fatal and is named by an orca_relay_process_fatal line first.
Every pooled client also keeps a permanent error listener. Fix level 2.
* fix(relay): name class 23 in the fence comment; move the fix-level bump to the read-timeout fix
R1 alone does not close the lost-reply stall, so the level rises with R2.
The fix-level comment now describes the fixed alert floor.
* fix(relay): survive and answer 503 for 25P03, the idle-in-transaction timeout a VM stall triggers
A whole-VM database stall of 6-7 s outlasts the 5 s idle-in-transaction
limit for a backend caught idle mid-transaction, and the server ends that
session with FATAL 25P03 on resume. The fence now survives that one class-25
code (the rest of class 25 stays fatal), and routes answer it as transient.
* style(relay): one transient SQLSTATE per line
* fix(relay): fence a socket reset or broken pipe the database layer rethrew
pg 8.22 rejects the active query with the raw socket error ('read
ECONNRESET'), which the message-only check let through to a fatal exit:
the 09-28 crash shape. Match ECONNRESET and EPIPE by code, gated by the
database-layer mark.
|
||
|
Jinwoo Hong
|
ad7af63b20 | fix(relay): throttle the connection-reservations autovacuum so a run stops evicting the shared database cache (#26370) | ||
|
Jinwoo Hong
|
3062b9d8d0 |
fix(relay): refuse host hellos fast when the database pool is timing them out; renewals jump the queue (#26362)
* fix(relay): refuse host hellos fast when the database pool is saturated; renewals jump the queue A reconnect herd (2026-10-06 18:34Z, 1,046 asia-east2 hosts dropped by the load balancer at once) queued hundreds of host hellos behind each cell's 16-connection pool. They timed out after 2 s, the desktops redialled into the same queue, and control-lease renewals starved behind them. - Host control upgrades get an immediate 503 (Retry-After: 2) once 32 callers wait for a pooled connection. Shipped desktops already treat that as a connect error and back off with jittered exponential retry. - General work now queues in PostgresPoolPressure instead of pg-pool, so renewal statements (priority lane) take the next released connection. - A renewal batch that never got a connection no longer fans out into one statement per host. - hostHellosShedDelta counts refusals in the runtime metrics. * fix(relay): shed host hellos on the pool's oldest wait, not its queue length asia-east2 queues of 50-196 waiters are routine and keep moving, so a 32-waiter limit would refuse hellos that were going to succeed. Shed only while the oldest pool waiter has waited 1.5 s of the 2 s acquire timeout, when a new hello would time out anyway; a refused desktop gets the same connect error and backoff it gets today, 2 s sooner. Rebinds over a live control (lease rotation) are never refused. * fix(relay): shed hellos only when a full pool queue has aged 1 s Per review: require both an oldest waiter of at least 1 s and a pool's worth of waiters, so one slow waiter cannot trip it, and state the real baseline (p99 of 2 waiters on 10-07). |
||
|
Jinwoo Hong
|
54ded3bc18 |
fix(relay): commit the cell counter in one round trip; cells boot without the database (#25765)
* fix(relay): commit the cell counter in one round trip; cells boot without the DB Step 2 (option B) cell image: - One-round-trip counter commit at acquireActivity, releaseActivity and activateControl: the final counter UPDATE and COMMIT go as one simple-query message. Server errors mean COMMIT never ran (retry as today; 22012 = no row, rolled back and disambiguated outside the transaction); a lost connection is never retried. - Cells skip the schema apply and region backfill, so they listen while the database is down and turn ready on their first successful query. - G13: rehome target connection headroom folded into the existing NOWAIT UPDATE, excluding the host's own reservation by key. - fixLevel on every runtime metrics line, plus declared (not applied) cell fix-level metrics and alert. - Per-desktop drain disconnect-gap measurement from existing log lines. - Census test that fails on floating database promises; fixes two shutdown sites. Lock-wait sample keeps the combined role. * fix(relay): make the outdated-image alert creatable: one PromQL condition, 1 h lookback, fixed floor A PromQL condition must be the only condition in its policy, and alerts on log-based metrics may look back at most 25 h. Replace the 6-day/7-day design with relay_cell_min_fix_level (tfvars, raised by a targeted apply after each wave) and one query: a serving cell below the floor or reporting no level, sustained 6 h. Drops the separate without-level metric. * fix(relay): review fixes: gap-script ordering, wider promise census, fused-path guard, row-busy as scheduled - Drain gap script: sort closes by time (gcloud exports newest first) and refuse an invalid drain start. - Census: any floating promise in relay src, including callback-discarded and never-read ones, with a reviewed never-rejects list. - Test the fused counter commit through the store the server builds, so a wrapper that stops forwarding commitWithFinal fails CI. - Same-cap shadow gate: a row-busy refusal (the host's own release still holds its row) is a scheduled 503, like an own early retry. No client change. * test(relay): judge drain redials by no host refused twice, not a refusal count The row-busy count tracks how many releases are still in flight at the dial (80 of 180 every run at 1 s, against a bar of 90). What matters is that the release has finished by the next dial: assert no host is refused twice, keep the time-to-placed p95 bound. * fix(relay): cap row-busy as scheduled at the drain-return admissions; bound the gap script's window Shadow gate: a row-busy refusal of a drained host follows its drain-return lane admission, so per minute only that many (plus a rounding margin of 2) are scheduled; the rest stay non-drain, so row contention the drain does not explain still fails the budget. Gap script: --drain-ended-at excludes the new container's closes after the roll; later grants still close a gap. |
||
|
Jinwoo Hong
|
52bd9afe9d |
refactor(relay): stop creating three tables nothing writes (#25354)
* refactor(relay): stop creating three tables nothing writes relay_confirmable_splices, relay_cell_drain_attempts and relay_migration_leases are created at every boot and referenced nowhere else on main except tests. In production all three hold 0 rows and pg_stat_user_tables shows 0 inserts, 0 sequential and 0 index scans since the 2026-09-28 stats reset. This removes them from the schema and the transaction-phase table, and drops the test references. It does not drop them: an older image still runs CREATE TABLE IF NOT EXISTS at boot, and a drop racing that create can fail the older boot's schema step. A follow-up can drop them once no image that creates them can be rolled back to. * docs(relay): note the account-erasure prerequisite for dropping retired tables |
||
 Neil
|
668fa71906 | Release SQLite transaction queues when startup fails (#22993) | ||
|
Jinwoo Hong
|
ae3d380b23 |
fix(relay): lock only the target cell row, last and NOWAIT, in the rehome commit (#22449)
* fix(relay): lock only the target cell row, last and NOWAIT, in the rehome commit The idle-rehome commit runs on the source cell. From an Asia cell each statement is a cross-region round trip, and the transaction locked every relay_cells row plus every runtime, capability and safety row before about twenty more statements, so each Asia-source rehome held the whole fleet's cell rows for ~3.6 s and every reconnect, renewal and placement queued or timed out behind it. The commit now reads the cell inventory and the runtime, capability and safety tables unlocked, keeps the control and worker rows locked (now NOWAIT), and takes one cell lock: the target row, in a single statement that locks it NOWAIT, re-checks enabled, general admission and capacity, and reserves the units, issued as the last statement before COMMIT. A target that changed admission, filled up, or is locked by another writer rolls the whole commit back and answers deferred (candidate-ineligible). The hold is sampled under a site label, so cellInventoryHoldMsMax still sees rehome holds and rehomeTargetRowHoldMsMax reports them apart. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * fix(relay): fail closed on the rehome target-row lock clause The target-row statement now carries FOR UPDATE ... NOWAIT unless the dialect is explicitly SQLite, so a wrapper that omits the optional dialect can no longer run the reservation unlocked. Test wrappers and the fault injection entry forward the dialect they wrap. The latency test also probes the admission and region tables at every round trip; only the target's admission row may be locked, and only before COMMIT. The runbook notes that an Asia-sourced commit holds the rehome control row for about 6.5 s, so a pause that fails once is retried. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb |
||
|
Jinwoo Hong
|
c8a5580659 |
fix(relay): re-place hosts off a cell isolated for a roll (#21911)
* fix(relay): re-place hosts off a cell isolated for a roll A roll isolates a cell by moving it out of the 'general' admission class; the cell then refuses every attach with 4503. The director never noticed, because the only liveness test it applies to a host's current cell reads `relay_cell_runtime.ready` and the heartbeat, and an isolated cell keeps heartbeating ready=1 for the whole drain. So every host on that cell was handed its own dead cell, closed, and handed it back — 500-1,900 hosts looping for 13-16 minutes per cell roll, at ~6 dials each per minute, with no neighbour absorbing anything. The sticky lane now treats a live incumbent whose admission is 'migration-only' — the state a roll's isolate step writes — the same way it treats a dead one: it returns null, which means "fall through to placement". The placement lane had the identical hole eleven lines further down, so it takes the same predicate; without that second swap the sticky change is inert, because placement would hand the pin straight back (a draining cell has more headroom than anyone). An isolated incumbent skips the dead-cell fence branch: that branch exists to prove an unreachable cell stopped serving a host, and this one is reachable and enforces the epoch itself. 'existing-only' is deliberately untouched — those cells serve the hosts they already hold, and only `assignmentStrandedOnUnservedCell` may release that pin. A host with an open `relay_assignment_migrations` row keeps its pin too, so this stays disjoint from the migration machinery. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * fix(relay): gate re-placement on a roll-isolation marker, not on admission Review of the first commit found the predicate wrong. `migration-only` is an admission class, not a drain signal: an Asia `--mode rollback`, an evacuation or forward-recovery target awaiting a separate promote dispatch, a failed same-cap wave's re-isolate, an abandoned migration retired on its target and a rehome settlement all park loaded cells there durably, with no migration lease and no open migration row. All five were indistinguishable from a roll's isolate, so the first commit would have converted `operate-relay-asia-admission --mode rollback` from a reversible admission flip into a mass move of ~4,000 hosts — and, because `leastLoadedCell` treated region as a preference, into us-central1. The signal is now an explicit stamp. `relay_cell_admission` gains a nullable `roll_isolated_at`, added through the shared schema runner's catalog pre-check so a migrated database takes no relation lock on boot and an un-migrated one gets a catalog-only rewrite. The same-cap isolate step is its only writer, via a new optional `rollIsolatedCells` on the selector apply; the same UPDATE that writes the state clears the stamp whenever a cell leaves 'migration-only', so a restore cannot leave one behind and a failed wave's re-isolate keeps the one it has. Every other admission writer omits the field, so its cells stay unmarked and their hosts stay pinned. Old directors ignore the field; old callers never send it. Region is now a constraint rather than a preference on this path only: a re-placement must find a general, live cell with connection headroom in the host's own region, or the pin is kept and one `orca_relay_sticky_replacement_deferred` event is logged. Cross-region spill is no longer reachable here. The fence bypass is narrowed to a live incumbent. It was always a no-op for the intended case, and for a stamped cell that stops heartbeating while still holding sockets it reopened split-brain; that cell now takes the dead-cell path unchanged. Also: the hot-path admission reader no longer throws on an unrecognised state — it sits on every sticky dial and the rule it feeds is "move the host", so an unreadable row has to mean "don't". And the sticky lane reads the admission row once for both the stranded rule and the stamp instead of twice. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * fix(relay): emit the re-placement events after the transaction commits CodeRabbit on assignment-store.ts:1075. Both events were written where they are decided, which is inside assignOnce's transaction. A reservation or lease write failing after that point rolls the placement back, but a line already on stdout cannot be rolled back with it — so the canary this PR asks an operator to read would count re-placements that never happened, and a Postgres transaction retry could leave a stale line behind as well. The transaction now returns its events alongside the RelayAssignment and the caller flushes them once it has resolved. Returning them rather than setting a variable in the enclosing scope is what makes the retry case safe too: only the attempt that committed can carry its events out. assign()'s signature is unchanged; the extra shape lives entirely inside assignOnce. orca_relay_sticky_replacement_deferred was moved the same way. It cost one more push into the array that already existed, and it is decided inside the same transaction, so leaving it behind would have been the odd case rather than the cheap one. The new test injects a failure on the first write after the decision, asserts no event is emitted, and asserts the assignment is still on its original cell — without that second assertion the absence would only prove the emit was early, not that it would have been wrong. A control dial with nothing injected emits exactly one event, so the case cannot pass on a broken harness. With the emit put back inside the transaction, it fails. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * fix(relay): expire the roll stamp, correct the wire note, assert the stamp landed Delta review findings B, D and E. A (the deferral path's cost) is deliberately not implemented; it is now written up under Follow-ups in the PR body as required before any Asia roll, because it cannot fire in a US canary. B, which also closes C: the stamp was written, carried and never compared to anything. A roll isolates and restores one cell inside ~15 minutes, so a stamp older than two hours is not a roll in progress. It is a failed wave whose failsafe re-isolated a possibly healthy cell and is waiting on an operator — the postmortem in this tree records gaps of hours — or an orphan left by a director rollback whose restore wrote 'general' without the clause that clears the stamp, which the selector's 'keep' branch would then preserve until some later park reactivated it. Both want the same answer and it is the pre-existing one: keep the pin. One comparison against a value already on the row. The bound takes the caller's `now` rather than reading the clock again, so one assign reasons about one instant; the stamp's age is now a thing that decides whether a host moves, and two clock reads could disagree across it. D: the comment beside the new request field claimed an updated caller reaching an older director "is simply ignored". The schema is .strict(), so it is a 400. That fails closed — the isolate aborts before MUTATION_STARTED is set and nothing is written — but it is a deploy ordering constraint, and it was undocumented. The comment now says so and the PR body's rollout notes carry it. E: nothing read the `rollIsolated` the script already prints, so an older script against a newer director would silently produce today's behaviour and the canary would read as "the fix did nothing" with no way to tell that from a wrong premise. Both isolate steps now assert it, beside the generation they already parse. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb |
||
|
Jinwoo Hong
|
eb6068a434 |
fix(relay): stop a terminated checked-out PostgreSQL client from killing the cell (#21840)
pg-pool removes its own `error` listener when it hands a client out (pg-pool@3.14.0 index.js:344) and only reattaches it in `_release` (index.js:385). Between acquire and release the client therefore has no `error` listener, so when Cloud SQL terminates that session mid-statement the emit becomes an unhandled 'error' event and the process exits. `absorbPostgresIdleClientErrors` cannot see it: pg-pool routes to `pool.on('error')` only from the idle listener. Attach a per-checkout `error` listener in the one seam every relay checkout passes through, log a single warn line, and release the client with the error so pg-pool destroys it instead of pooling a dead connection. The listener is removed on release so it cannot accumulate. The in-flight query still rejects, so existing failure reporting and the transaction retry ladder are unchanged. Claude-Session: https://claude.ai/session/ced32ebb-7155-4413-adad-1eccd14c2010 |
||
|
Jinwoo Hong
|
fa0010e8d6 |
fix(relay): abort rehomes whose host never arrived, without disabling the switch (#21813)
A regional rehome whose host went offline right after accepting the move left its migration row open forever: the target had registered it, the host held nothing on the source, and the completion sweep could never finish it. Eight such rows filled REGIONAL_REHOME_CONCURRENT_LIMIT and every later candidate came back deferred, silently, for 21 hours. The only sweep that touched them fires at 24 hours and also sets enabled = 0 on the durable control, so the first leak to age out would have turned rehoming off, repeatedly. Adds a director sweep that rolls such an attempt back to its source after one migration lease, with abort_reason = 'host_not_arrived', reusing the existing rollback (assignment epoch bump back to the source, lease removal, superseded target reservation release) and leaving the switch untouched. The 24-hour sweep keeps its disable as a last-resort latch. The source cell now names why it deferred, on a new optional response field, and the director stops walking its candidate page on a deferral no later candidate can pass. Each poll that dispatched logs one summary line. Claude-Session: https://claude.ai/session/ced32ebb-7155-4413-adad-1eccd14c2010 |
||
|
Jinwoo Hong
|
ce5d8c02d4 |
fix(relay): wait out a cold proxy at boot instead of exiting the cell (#21516)
* fix(relay): wait out a cold proxy at boot instead of exiting the cell A cell container starts its relay process beside a cloud-sql-proxy that is itself still dialling. The first pool acquire therefore competes with a proxy cold start, and the 2s connect timeout that protects the request path fires before the proxy is listening. `openRelayDatabase` rejects out of the region backfill, the top-level await rejects, and the process exits; COS restarts the container and the next boot succeeds 1-3s later. The 2026-09-18 fleet roll saw 0-7 of these per cell, including on cells with zero hosts, so it is a property of the boot sequence rather than of database load. The boot open now retries on transient errors only, inside a 45s wall-clock window with exponential backoff from 250ms to 4s. The classifier is the one the request path already uses, so a rejected credential or a bad URL still exits on the first attempt. Each wait logs `orca_relay_boot_database_retry` and a give-up logs `orca_relay_boot_database_failed`, both with the bounded error category, so a rollout can tell a slow boot from a stuck one without reading container exit codes. The bounded startup retry is lifted out of `reconcileCellAdmissionAtStartup`, which had the same loop; its attempt budget, flat delay, and both log events are unchanged (a flat delay is a cap equal to the base). * fix(relay): retry the boot open only when Postgres is unreachable The boot open re-runs the schema apply, and applyPostgresSchema refuses to repeat a DDL lock timeout on purpose: relation locks are granted in queue order, so a repeat parks every writer behind the same statement again. Gating the boot retry on the full request-path classifier would have re-queued it up to 16 times in 45s on sustained 55P03 - the mechanism behind the 2026-09-16 outage. The boot call site now has its own predicate: pool connect failures (both connect-timeout messages and an acquire-marked early-ended socket) plus 08001 and 08006. Lock and overload SQLSTATEs - 55P03, 57014, 53300 - exit on the first attempt. The retry predicate moves onto the policy because what a step re-runs, not the request path, decides what it may repeat; the startup reconcile keeps the full classifier, which is what lets it wait out 55P03. |
||
|
Jinwoo Hong
|
ddbad2218b |
chore(relay): drop the live-basis partial index that the planner never picks (#21305)
Added in #21301 on the reasoning that the composite (active, deadline) index spans all 6.65M rows to find a few hundred live ones. Measured post-merge against production-shaped history, that reasoning does not hold: a basis is inserted active and flipped to 0, so the partial index accumulates one dead entry per deactivation exactly as the composite one does. Scan buffers are identical to composite-only in every state, 11,099 cold, 2,522 warm, 9 after VACUUM, and the planner picks the composite index throughout. The extra index costs ~65 bytes of WAL per basis insert, about 22% more. The relief is the reaper plus vacuum, which #21301 already ships. Removes the statement from SCHEMA, restores the plan test to pinning the composite index by name, and records why a narrower index is not the cure so the next reader does not re-derive it. |
||
|
Jinwoo Hong
|
cbd04704d6 |
perf(relay): index and gate the credential cleanup sweeps that seq-scan relay_invites (#21301)
* perf(relay): index and gate the credential cleanup sweeps that seq-scan relay_invites The credential cleanup ran every 30s in all 23 cells as well as the director. Both of its relay_invites passes matched columns no index covered, so each one seq-scanned the whole table inside the maintenance transaction: 56 calls/min fleet-wide, 129ms and 63ms typical and 57s at the tail, to return about one row every nine minutes. Adds partial indexes matching each sweep predicate, gives the cleanup the same owner as the assignment sweep, and reaps terminal invites after seven days so the table stops growing for the life of the database. Every index carries the schema-deferrable marker: an operator builds them with CREATE INDEX CONCURRENTLY, and the catalog pre-check skips them from then on. * perf(relay): index live bases and reap settled connection authorizations relay_connection_bases is the dominant cost in the cleanup transaction: 195 ms of the 268 ms average, with ~5,800 shared buffer hits per call even though it already uses relay_connection_bases_active_deadline. That index spans all 6.65M rows, and only a few hundred are ever active. Adds a partial index on the live rows alone, and reaps settled rows from relay_connection_bases and relay_direct_authorizations once their deadline is more than a day past. Both readers of either table require the row active/unconsumed and inside its deadline, and every deadline is set at most 30s past insert, so a settled row can never authorize anything again. The composite index stays: it is the only one covering active = 0, and it is what lets the drained reaper learn there is nothing to do from the index rather than the 1.5 GB heap. Measured at 200k rows, 5 buffers with it and 1,274 without. * test(relay): accept either bases index in the sweep plan assertion The negative assertion pinned a planner choice rather than the invariant: either index keeps the sweep off the 1.5 GB heap, and which one wins on cost is not something the test should fix. Matches how the same file already handles the two invite sweep indexes. Also names the column the authorization reaper actually measures, which is consumed_at rather than deadline. |
||
|
Jinwoo Hong
|
8e8a9b38ea |
perf(relay): stop indexing the column every control renewal writes (#21286)
* perf(relay): stop indexing the column every control renewal writes relay_assignment_activity_expiry indexes expires_at on relay_assignment_activity_leases, and expires_at is what every control renewal updates: ~471 calls/s, all of them non-HOT because a changed indexed column forbids HOT. The index has one reader, the 30s expiry sweep, which seq-scans the whole 14.8k-row table in under a millisecond. Drop it, and set fillfactor to 70 so a renewal has room for a second row version on its own page. Measured on postgres:16-alpine over 14.8k rows, WAL bytes per renewal and HOT ratio: index, fillfactor 100 (today) 0% HOT 371 B index, fillfactor 70 0% HOT 246 B no index, fillfactor 100 0.5% HOT 298 B no index, fillfactor 70 100% HOT 80 B Both are needed: the index makes HOT illegal, and the default fillfactor leaves no page space to make it possible. Neither statement can use the catalog pre-check as it stood. DROP INDEX IF EXISTS resolves the name before it locks, so once the index is gone it costs a catalog miss and takes no lock on the table - pinned in the lock-target census as the one exempt statement. ALTER TABLE SET does take a lock, so it gets a new 'reloption' target kind that asks pg_class.reloptions for the name=value pair, keeping the invariant that no lock-taking statement reaches a warm boot unchecked. * fix(relay): pre-check the activity-expiry drop and let it defer on a lock timeout The drop had no catalog pre-check, so it was sent on every boot, and a 55P03 from it was fatal: apply-postgres-schema throws on a lock timeout with no retry. On the migration boot that combination is a crash loop. All 28 directors reach the same DROP INDEX at once, it needs ACCESS EXCLUSIVE on a table written ~475/s with lock_timeout at 1s, and a boot that fails restarts the instance to re-queue the same DDL behind the same writers. Two changes: - A new 'index-by-name' lock target. A DROP INDEX names no table, so the existing index check could not serve it; this one resolves by name through the search_path with relkind = 'i', which is how the DROP itself resolves, and skips when absent. DROP INDEX now counts as lock-taking in the census, so it is covered rather than exempt, and IF EXISTS is required the way it is on DROP CONSTRAINT. - A 'schema-deferrable' marker, read from a statement's leading comment. A 55P03 on a marked statement logs orca_relay_postgres_schema_object_deferred and leaves the statement unapplied instead of failing the boot; the next boot re-sends it. Both activity-lease migrations carry it. Everything else keeps the old contract and still fails loudly. SchemaApplySummary gains a deferred count so a boot that skipped work is distinguishable from one with nothing to do. Verified against a real server: with the index present and the table held in ACCESS EXCLUSIVE by another session, both statements defer, the boot completes, nothing is half-applied, and the next boot finishes the job. A warm boot now sends neither statement at all. |
||
|
Jinwoo Hong
|
a3046cd27b |
fix(relay): treat database pool connect failures as transient, not director faults (#21243)
* fix(relay): treat pool connect failures as transient, not director faults pg-pool raises connection-acquire failures as a plain Error with no SQLSTATE, so the transient classifier matched only one of the three messages it can produce. The other two reached the routes unclassified and became HTTP 500s, which is what the rollout safety gate counts. The acquire boundary now marks the errors it produces, so "Connection terminated unexpectedly" counts as transient when the socket died during the handshake and stays a hard failure mid-statement, where a retry could repeat a commit whose outcome is unknown. /v1/regions and /v1/admin/evacuation-status gain the transient handling /v1/assign and /v1/resolve already had. * fix(relay): mirror the pool-connect verdict in failure diagnostics The query-failure event's connectionTimeout boolean matched one of the two messages connectionTimeoutMillis can produce, so the 210 dialling timeouts in the last day logged as false and were invisible to the field meant to find them. The pool-connect vocabulary now lives beside the acquire boundary that owns it, and both the router's classifier and the diagnostics read it from there, so the two cannot drift. The event also carries the routing verdict the caller already computed, making "how much of this burst reached users as a 500" one field. * fix(relay): null-safe transient classification and honest transient docs The classifier now runs inside the query catch, where a thrown null or undefined would have turned a database failure into a TypeError that buried it. The diagnostics doc claimed transient maps to a 503 or a 500. Sweeps, startup reconciliation, and admin routes that answer 409 all emit the same event, so counting the false ones over-states user-facing hard failures. |
||
|
Jinwoo Hong
|
0699d73fd6 |
fix(relay): skip boot-time DDL when the catalog already has the object (#21147)
* fix(relay): skip boot-time DDL when the catalog already has the object CREATE INDEX IF NOT EXISTS and ALTER TABLE ADD COLUMN IF NOT EXISTS take their relation lock before the server evaluates the existence test, so a boot on an already-migrated database still joins the lock queue. Relation locks are granted in queue order, so every writer queues behind it. The shared runner now asks pg_catalog whether the index or column is already there and skips the statement when a row comes back, and 55P03 is no longer retried by default: with the pre-check ahead of it, a lock timeout means the object is genuinely missing and each retry re-enters the queue. Push keeps the old retry behind an explicit option. * fix(relay): tie the index pre-check to its table and fail on an unreadable target Three defects found in review of the auth reference implementation: - The catalog query matched an index by name inside the table's namespace without checking it belonged to that table. Index names are unique per schema, not per table, so a same-named index on a sibling table answered yes and the real index was skipped forever. Added i.indrelid = t.oid. - Lock-target derivation read a keyword sitting in an identifier position as the object name: CREATE UNIQUE INDEX CONCURRENTLY ON t(c) yielded the name CONCURRENTLY, and ADD COLUMN IF NOT EXISTS with no column yielded IF. A wrong target is worse than none, so keywords are now excluded and an index or column statement whose target cannot be read throws at boot with the statement text instead of falling through to the lock path. - A concurrent-create collision retried the CREATE INDEX, taking SHARE on the table again for an object another director had just finished creating. The catalog is re-asked instead and a present object counts as skipped. * fix(relay): pre-check constraint swaps so a warm boot sends no DDL at all The two ALTER TABLE constraint statements were the last lock-taking statements without a pre-check, so every boot still took ACCESS EXCLUSIVE on relay_region_rehome_attempts twice. A lock target now carries the catalog answer that means there is nothing left to do. ADD CONSTRAINT skips when pg_constraint already names it; DROP CONSTRAINT IF EXISTS is the inverse and skips when it does not, because nothing to drop is nothing to do. The match is by name only: the CHECK body is generated from RELAY_REGIONS, so comparing it would re-run the swap on every region change. Changing a definition under the same name is an operator migration, and the rule comment beside SCHEMA says so. A bare DROP CONSTRAINT gets no target and throws at boot, because skipping it would swallow the undefined_object the server is supposed to raise. The census invariant is now that every lock-taking statement has a pre-check, with no exceptions, and the warm-boot Postgres test asserts zero statements sent rather than two. * fix(relay): refuse a multi-action ALTER TABLE instead of pre-checking its first action `ALTER TABLE t ADD COLUMN IF NOT EXISTS a TEXT, ADD COLUMN IF NOT EXISTS b TEXT` derived the target for `a` alone, so once `a` existed the whole statement was skipped and `b` was never added. The first subcommand parses, so neither the parse throw nor the census caught it. A lock-taking ALTER TABLE with a comma outside parentheses, quotes and comments now throws at boot. One action per statement, or no pre-check is possible. Commas inside a parenthesised type, a CHECK body, a quoted default or a comment are unaffected, and push's 18 statements still parse. * fix(relay): strip every comment before classifying, fold catalog names, count brackets Four findings from the bot reviews on #21147: - A comment between two keywords (ALTER TABLE t ADD /* note */ COLUMN c TEXT) was invisible to both the classification regexes and the must-parse shapes, so the statement got no target AND no throw and ran with no pre-check. Every comment is now stripped quote-aware before classification, nested block comments included. The server is still sent the original text. - hasTopLevelComma counted parentheses but not square brackets, so ADD COLUMN c bigint[] DEFAULT ARRAY[1, 2] read as two subcommands and failed the boot. - bareIdentifier split a qualified name on '.' regardless of quoting, so "a.b" became b", and it kept the written case while Postgres folds an unquoted identifier to lower case before storing it in relname, attname and conname. The name is now tokenised quote-aware and folded, with the qualified table text still passed to to_regclass as written. - sqlWithoutLeadingComments is renamed sqlWithoutComments to match. Relay's 74 statements and push's 18 all still parse, and no relay target name changed: every identifier there was already lower case. * fix(relay): treat a dollar-quoted body as opaque in both scanners A comment marker, comma, parenthesis or bracket inside `$$...$$` or `$tag$...$tag$` is text. The closing delimiter has to match the opening tag exactly, so an inner `$$` inside a `$tag$` body is more text rather than the end, and a tag cannot start with a digit, which keeps a `$1` placeholder from reading as an opener. Relay's pg_stat_statements DO block is the only dollar-quoted statement in the schema, and it now survives the stripper byte-identical. A test asserts that against the real statement. |
||
|
Jinwoo Hong
|
71e308e574 |
feat(relay): count failed cell-inventory lock acquisitions (#21067)
* feat(relay): count failed cell-inventory lock acquisitions The cell inventory lock is taken NOWAIT, so contention errors with 55P03 and retries instead of waiting. CellInventoryHoldSamples.record only runs after a successful acquisition, so the hold metrics were structurally blind to the dominant failure mode: production showed ~65 failed fleet-wide acquisitions per minute while cellInventoryHoldMsMax read a benign 53ms mean. Count failures next to the holds and publish them as cellInventoryLockUnavailable in orca_relay_runtime_metrics. Drained on both the commit and the rollback path, since a 55P03 rolls its transaction back. * fix(relay): separate request-path lock timeouts from sweep deferrals Review caught that the first counter only incremented under failIfUnavailable, which is the sweep mode. Background sweeps take the inventory NOWAIT and re-derive a skipped candidate next tick, so those deferrals are by design and already reported as orca_relay_sweep_cell_inventory_busy. The request path uses a bounded lock_timeout instead, whose expiry raises the same 55P03 without NOWAIT and was not counted at all -- so the metric measured only the benign population and missed the user-visible one. Split them: cellInventoryLockUnavailable for NOWAIT deferrals, cellInventoryLockTimeouts for expired bounded waits. Production over 30 minutes shows why the distinction matters -- roughly 1,200 fleet-wide sweep deferrals against roughly 10/min request-path timeouts. Adds transaction-path coverage for both drains, which were previously unpinned. Timeouts count per attempt, not per request, since 55P03 is retryable. * fix(relay): publish the cell-inventory lock metrics to Cloud Monitoring google_logging_metric.relay_snapshot only creates metrics for fields listed in relay_runtime_metrics, and the cellInventoryHold* fields were never added when the hold telemetry landed. They have been log-only since, so nothing could alert on the lock and the contention stayed invisible in exactly the way the telemetry was meant to prevent. Maps the three hold fields and both new failure counters. Also corrects the field comment: the split is by wait policy, not by caller. assignOnce takes the inventory fail-fast on its first placement attempt, so request-reachable sites land in cellInventoryLockUnavailable too; that lane reads as contention pressure, and the expired bounded wait is the stall lane. |
||
|
Jinwoo Hong
|
b07c4032ea | Log bounded PostgreSQL acquisition and execution failure diagnostics (#20749) | ||
|
Jinwoo Hong
|
d51747e4c4 | feat(relay): expose preloaded PostgreSQL statement statistics (#20712) | ||
|
Jinwoo Hong
|
cd9aa43a2c |
feat(relay): correct regional placement only when the source is idle (#20105)
* feat(relay): correct regional placement only at an idle source * test(relay): lock source activity capacity semantics |
||
|
Jinwoo Hong
|
f5be177e44 |
fix(relay): rehome hosts to their preferred region in either direction (#19241)
* fix(relay): rehome hosts to their preferred region in either direction The regional-rehome worker only moved hosts from a us-central1 cell to an asia-east2 one, so a host whose desktop later records us-central1 stays where it was put. Rehoming now compares the fresh preference against the region of the cell the host is on and moves it to a general cell in the preferred region either way, through the same drain, migrate, safety, and rate-limit machinery. - relay_region_rehome_attempts.preferred_region accepts both regions; existing databases are upgraded in place by an idempotent named-constraint swap that is safe when several directors start at once. - A target must carry the drain protocol too: moving a host onto a cell it can never be drained off again is the trap this change exists to undo. The fleet whose health gates a rehome is now every general drainable cell, which is exactly the set of legal sources and targets. - The trust probe accepts a source cell in any region. No wire change, and no behaviour change while the durable control is off. * fix(relay): bound bidirectional rehoming with a per-host cooldown Moving hosts in both directions removed the property that made the old one-way worker self-terminating: a desktop whose region probe flips would be dragged back and forth, one full drain and migrate per flip, because the preference age never expires while the host keeps reconnecting. - relay_region_rehome_control gains host_cooldown_ms, an operator input plumbed like preference_max_age_ms (workflow, ops script, admin route, durable row) and defaulted to seven days. A host with any attempt row inside the window, whichever way that move went, is not a candidate; the claim re-reads it under lock so an attempt landing between scan and claim cannot start a second move. Skips are named host_cooldown, and the lookup rides a new index on (user_id, relay_host_id, created_at). - The candidate scan now also requires the target cell to be enabled, so it mirrors the claim-time filter exactly and stops spending batch slots on candidates that are certain to be skipped. - Region CHECK lists are rendered from the shared region list instead of being written out four times. - The operations runbook states that cells without the drain protocol are neither sources, targets, nor members of the safety gate. * fix(relay): keep rehome reads and brakes working across the cooldown rollout The ops script validated hostCooldownMs on every inspected control, so against any director image predating the field inspect, pause, disable, and failed-enable recovery all threw client-side. The workflow always runs from main while the director image is operator-supplied, so that window opened at merge and reopened on every rollback: the operator lost read-only visibility and both emergency brakes while the worker could still be enabled. The field is now validated only when the director reports it, and every apply body that echoes an inspected control omits the key when that control lacks it, so a legacy director never sees an unknown key. The write path stays fail-closed the other way: enable refuses up front, before any mutation, when the director does not report a cooldown it could honour. Also replaces two bare 'us-central1' defaults with RELAY_DEFAULT_REGION. |
||
|
Jinwoo Hong
|
b6f453df06 |
perf(relay): per-cell inventory locks, delta counters, and a pool statement timeout (#18722)
The sticky refresh path and reservation reconciliation both took the fleet-wide `relay_cells ... FOR UPDATE` scan to mutate one or two rows, so one busy cell queued unrelated reconnects and migration completions behind it. Both now lock only the rows they touch, in the same ascending cell_id order, and the sticky grant moves its counter by a delta instead of writing back a snapshot value. Placement keeps the ordered inventory lock: choosing the least-loaded cell is a genuinely fleet-wide decision, and dynamically locking only the selected target is what allowed cross-cell cycles before. The pool's statement_timeout becomes env-configurable and a 57014 now reaches the bounded transaction retry instead of surfacing as a terminal failure. Schema DDL moves to its own `max: 1`, statement_timeout-free pool that is ended before the serving pool opens, so a slow CREATE INDEX cannot inherit a request deadline it will never fit inside. |
||
|
Jinwoo Hong
|
d66386bc82 |
fix(cloud): bound and yield the relay's global cell-inventory lock (#18521)
Mirrors stablyai/orca-cloud#471 (squash c3354e8), byte-identical under cloud/. The relay's cell-inventory lock (SELECT ... FROM relay_cells FOR UPDATE over all 23 rows) is one global critical section shared by the assignment hot path and every director sweep; with the pool's 1s lock_timeout a blocked waiter held a pooled client for a full second, producing ~690 55P03 retries per 5 minutes in production. Request paths now bound the wait at 500ms with a SET LOCAL that is restored to the pool default before the next statement; director-only sweeps take the lock NOWAIT and skip the tick; sweep timers are jittered; hold time is exported as additive runtime-metrics fields so the bound can be tuned. |
||
|
Jinwoo Hong
|
3eec77c11a |
chore(cloud): add the relay fence broker, ops console, Terraform root, scripts, and 24 cloud-* workflows (#18413)
Phase 6 of the relay split: the relay's deploy/operate surface moves under cloud/ with 24 cloud-* workflows gated on ORCA_CLOUD_OPERATIONS_ENABLED, the Cloud SQL rollout lease action, the relay Terraform root (dual-accept identities for both repositories), scripts, docs, CODEOWNERS, and a terraform validate job in Cloud Verify. |