mirror of
https://github.com/stablyai/orca.git
synced 2026-10-20 00:03:45 +00:00
8d516cc28ee8b770e8f5d602eaeefc5840f39b9f
19
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Neil
|
668fa71906 | Release SQLite transaction queues when startup fails (#22993) | ||
 Jinwoo Hong
|
ae3d380b23 |
fix(relay): lock only the target cell row, last and NOWAIT, in the rehome commit (#22449)
* fix(relay): lock only the target cell row, last and NOWAIT, in the rehome commit The idle-rehome commit runs on the source cell. From an Asia cell each statement is a cross-region round trip, and the transaction locked every relay_cells row plus every runtime, capability and safety row before about twenty more statements, so each Asia-source rehome held the whole fleet's cell rows for ~3.6 s and every reconnect, renewal and placement queued or timed out behind it. The commit now reads the cell inventory and the runtime, capability and safety tables unlocked, keeps the control and worker rows locked (now NOWAIT), and takes one cell lock: the target row, in a single statement that locks it NOWAIT, re-checks enabled, general admission and capacity, and reserves the units, issued as the last statement before COMMIT. A target that changed admission, filled up, or is locked by another writer rolls the whole commit back and answers deferred (candidate-ineligible). The hold is sampled under a site label, so cellInventoryHoldMsMax still sees rehome holds and rehomeTargetRowHoldMsMax reports them apart. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * fix(relay): fail closed on the rehome target-row lock clause The target-row statement now carries FOR UPDATE ... NOWAIT unless the dialect is explicitly SQLite, so a wrapper that omits the optional dialect can no longer run the reservation unlocked. Test wrappers and the fault injection entry forward the dialect they wrap. The latency test also probes the admission and region tables at every round trip; only the target's admission row may be locked, and only before COMMIT. The runbook notes that an Asia-sourced commit holds the rehome control row for about 6.5 s, so a pause that fails once is retried. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb |
||
|
Jinwoo Hong
|
c8a5580659 |
fix(relay): re-place hosts off a cell isolated for a roll (#21911)
* fix(relay): re-place hosts off a cell isolated for a roll A roll isolates a cell by moving it out of the 'general' admission class; the cell then refuses every attach with 4503. The director never noticed, because the only liveness test it applies to a host's current cell reads `relay_cell_runtime.ready` and the heartbeat, and an isolated cell keeps heartbeating ready=1 for the whole drain. So every host on that cell was handed its own dead cell, closed, and handed it back — 500-1,900 hosts looping for 13-16 minutes per cell roll, at ~6 dials each per minute, with no neighbour absorbing anything. The sticky lane now treats a live incumbent whose admission is 'migration-only' — the state a roll's isolate step writes — the same way it treats a dead one: it returns null, which means "fall through to placement". The placement lane had the identical hole eleven lines further down, so it takes the same predicate; without that second swap the sticky change is inert, because placement would hand the pin straight back (a draining cell has more headroom than anyone). An isolated incumbent skips the dead-cell fence branch: that branch exists to prove an unreachable cell stopped serving a host, and this one is reachable and enforces the epoch itself. 'existing-only' is deliberately untouched — those cells serve the hosts they already hold, and only `assignmentStrandedOnUnservedCell` may release that pin. A host with an open `relay_assignment_migrations` row keeps its pin too, so this stays disjoint from the migration machinery. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * fix(relay): gate re-placement on a roll-isolation marker, not on admission Review of the first commit found the predicate wrong. `migration-only` is an admission class, not a drain signal: an Asia `--mode rollback`, an evacuation or forward-recovery target awaiting a separate promote dispatch, a failed same-cap wave's re-isolate, an abandoned migration retired on its target and a rehome settlement all park loaded cells there durably, with no migration lease and no open migration row. All five were indistinguishable from a roll's isolate, so the first commit would have converted `operate-relay-asia-admission --mode rollback` from a reversible admission flip into a mass move of ~4,000 hosts — and, because `leastLoadedCell` treated region as a preference, into us-central1. The signal is now an explicit stamp. `relay_cell_admission` gains a nullable `roll_isolated_at`, added through the shared schema runner's catalog pre-check so a migrated database takes no relation lock on boot and an un-migrated one gets a catalog-only rewrite. The same-cap isolate step is its only writer, via a new optional `rollIsolatedCells` on the selector apply; the same UPDATE that writes the state clears the stamp whenever a cell leaves 'migration-only', so a restore cannot leave one behind and a failed wave's re-isolate keeps the one it has. Every other admission writer omits the field, so its cells stay unmarked and their hosts stay pinned. Old directors ignore the field; old callers never send it. Region is now a constraint rather than a preference on this path only: a re-placement must find a general, live cell with connection headroom in the host's own region, or the pin is kept and one `orca_relay_sticky_replacement_deferred` event is logged. Cross-region spill is no longer reachable here. The fence bypass is narrowed to a live incumbent. It was always a no-op for the intended case, and for a stamped cell that stops heartbeating while still holding sockets it reopened split-brain; that cell now takes the dead-cell path unchanged. Also: the hot-path admission reader no longer throws on an unrecognised state — it sits on every sticky dial and the rule it feeds is "move the host", so an unreadable row has to mean "don't". And the sticky lane reads the admission row once for both the stranded rule and the stamp instead of twice. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * fix(relay): emit the re-placement events after the transaction commits CodeRabbit on assignment-store.ts:1075. Both events were written where they are decided, which is inside assignOnce's transaction. A reservation or lease write failing after that point rolls the placement back, but a line already on stdout cannot be rolled back with it — so the canary this PR asks an operator to read would count re-placements that never happened, and a Postgres transaction retry could leave a stale line behind as well. The transaction now returns its events alongside the RelayAssignment and the caller flushes them once it has resolved. Returning them rather than setting a variable in the enclosing scope is what makes the retry case safe too: only the attempt that committed can carry its events out. assign()'s signature is unchanged; the extra shape lives entirely inside assignOnce. orca_relay_sticky_replacement_deferred was moved the same way. It cost one more push into the array that already existed, and it is decided inside the same transaction, so leaving it behind would have been the odd case rather than the cheap one. The new test injects a failure on the first write after the decision, asserts no event is emitted, and asserts the assignment is still on its original cell — without that second assertion the absence would only prove the emit was early, not that it would have been wrong. A control dial with nothing injected emits exactly one event, so the case cannot pass on a broken harness. With the emit put back inside the transaction, it fails. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * fix(relay): expire the roll stamp, correct the wire note, assert the stamp landed Delta review findings B, D and E. A (the deferral path's cost) is deliberately not implemented; it is now written up under Follow-ups in the PR body as required before any Asia roll, because it cannot fire in a US canary. B, which also closes C: the stamp was written, carried and never compared to anything. A roll isolates and restores one cell inside ~15 minutes, so a stamp older than two hours is not a roll in progress. It is a failed wave whose failsafe re-isolated a possibly healthy cell and is waiting on an operator — the postmortem in this tree records gaps of hours — or an orphan left by a director rollback whose restore wrote 'general' without the clause that clears the stamp, which the selector's 'keep' branch would then preserve until some later park reactivated it. Both want the same answer and it is the pre-existing one: keep the pin. One comparison against a value already on the row. The bound takes the caller's `now` rather than reading the clock again, so one assign reasons about one instant; the stamp's age is now a thing that decides whether a host moves, and two clock reads could disagree across it. D: the comment beside the new request field claimed an updated caller reaching an older director "is simply ignored". The schema is .strict(), so it is a 400. That fails closed — the isolate aborts before MUTATION_STARTED is set and nothing is written — but it is a deploy ordering constraint, and it was undocumented. The comment now says so and the PR body's rollout notes carry it. E: nothing read the `rollIsolated` the script already prints, so an older script against a newer director would silently produce today's behaviour and the canary would read as "the fix did nothing" with no way to tell that from a wrong premise. Both isolate steps now assert it, beside the generation they already parse. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb |
||
|
Jinwoo Hong
|
eb6068a434 |
fix(relay): stop a terminated checked-out PostgreSQL client from killing the cell (#21840)
pg-pool removes its own `error` listener when it hands a client out (pg-pool@3.14.0 index.js:344) and only reattaches it in `_release` (index.js:385). Between acquire and release the client therefore has no `error` listener, so when Cloud SQL terminates that session mid-statement the emit becomes an unhandled 'error' event and the process exits. `absorbPostgresIdleClientErrors` cannot see it: pg-pool routes to `pool.on('error')` only from the idle listener. Attach a per-checkout `error` listener in the one seam every relay checkout passes through, log a single warn line, and release the client with the error so pg-pool destroys it instead of pooling a dead connection. The listener is removed on release so it cannot accumulate. The in-flight query still rejects, so existing failure reporting and the transaction retry ladder are unchanged. Claude-Session: https://claude.ai/session/ced32ebb-7155-4413-adad-1eccd14c2010 |
||
|
Jinwoo Hong
|
fa0010e8d6 |
fix(relay): abort rehomes whose host never arrived, without disabling the switch (#21813)
A regional rehome whose host went offline right after accepting the move left its migration row open forever: the target had registered it, the host held nothing on the source, and the completion sweep could never finish it. Eight such rows filled REGIONAL_REHOME_CONCURRENT_LIMIT and every later candidate came back deferred, silently, for 21 hours. The only sweep that touched them fires at 24 hours and also sets enabled = 0 on the durable control, so the first leak to age out would have turned rehoming off, repeatedly. Adds a director sweep that rolls such an attempt back to its source after one migration lease, with abort_reason = 'host_not_arrived', reusing the existing rollback (assignment epoch bump back to the source, lease removal, superseded target reservation release) and leaving the switch untouched. The 24-hour sweep keeps its disable as a last-resort latch. The source cell now names why it deferred, on a new optional response field, and the director stops walking its candidate page on a deferral no later candidate can pass. Each poll that dispatched logs one summary line. Claude-Session: https://claude.ai/session/ced32ebb-7155-4413-adad-1eccd14c2010 |
||
|
Jinwoo Hong
|
ce5d8c02d4 |
fix(relay): wait out a cold proxy at boot instead of exiting the cell (#21516)
* fix(relay): wait out a cold proxy at boot instead of exiting the cell A cell container starts its relay process beside a cloud-sql-proxy that is itself still dialling. The first pool acquire therefore competes with a proxy cold start, and the 2s connect timeout that protects the request path fires before the proxy is listening. `openRelayDatabase` rejects out of the region backfill, the top-level await rejects, and the process exits; COS restarts the container and the next boot succeeds 1-3s later. The 2026-09-18 fleet roll saw 0-7 of these per cell, including on cells with zero hosts, so it is a property of the boot sequence rather than of database load. The boot open now retries on transient errors only, inside a 45s wall-clock window with exponential backoff from 250ms to 4s. The classifier is the one the request path already uses, so a rejected credential or a bad URL still exits on the first attempt. Each wait logs `orca_relay_boot_database_retry` and a give-up logs `orca_relay_boot_database_failed`, both with the bounded error category, so a rollout can tell a slow boot from a stuck one without reading container exit codes. The bounded startup retry is lifted out of `reconcileCellAdmissionAtStartup`, which had the same loop; its attempt budget, flat delay, and both log events are unchanged (a flat delay is a cap equal to the base). * fix(relay): retry the boot open only when Postgres is unreachable The boot open re-runs the schema apply, and applyPostgresSchema refuses to repeat a DDL lock timeout on purpose: relation locks are granted in queue order, so a repeat parks every writer behind the same statement again. Gating the boot retry on the full request-path classifier would have re-queued it up to 16 times in 45s on sustained 55P03 - the mechanism behind the 2026-09-16 outage. The boot call site now has its own predicate: pool connect failures (both connect-timeout messages and an acquire-marked early-ended socket) plus 08001 and 08006. Lock and overload SQLSTATEs - 55P03, 57014, 53300 - exit on the first attempt. The retry predicate moves onto the policy because what a step re-runs, not the request path, decides what it may repeat; the startup reconcile keeps the full classifier, which is what lets it wait out 55P03. |
||
|
Jinwoo Hong
|
ddbad2218b |
chore(relay): drop the live-basis partial index that the planner never picks (#21305)
Added in #21301 on the reasoning that the composite (active, deadline) index spans all 6.65M rows to find a few hundred live ones. Measured post-merge against production-shaped history, that reasoning does not hold: a basis is inserted active and flipped to 0, so the partial index accumulates one dead entry per deactivation exactly as the composite one does. Scan buffers are identical to composite-only in every state, 11,099 cold, 2,522 warm, 9 after VACUUM, and the planner picks the composite index throughout. The extra index costs ~65 bytes of WAL per basis insert, about 22% more. The relief is the reaper plus vacuum, which #21301 already ships. Removes the statement from SCHEMA, restores the plan test to pinning the composite index by name, and records why a narrower index is not the cure so the next reader does not re-derive it. |
||
|
Jinwoo Hong
|
cbd04704d6 |
perf(relay): index and gate the credential cleanup sweeps that seq-scan relay_invites (#21301)
* perf(relay): index and gate the credential cleanup sweeps that seq-scan relay_invites The credential cleanup ran every 30s in all 23 cells as well as the director. Both of its relay_invites passes matched columns no index covered, so each one seq-scanned the whole table inside the maintenance transaction: 56 calls/min fleet-wide, 129ms and 63ms typical and 57s at the tail, to return about one row every nine minutes. Adds partial indexes matching each sweep predicate, gives the cleanup the same owner as the assignment sweep, and reaps terminal invites after seven days so the table stops growing for the life of the database. Every index carries the schema-deferrable marker: an operator builds them with CREATE INDEX CONCURRENTLY, and the catalog pre-check skips them from then on. * perf(relay): index live bases and reap settled connection authorizations relay_connection_bases is the dominant cost in the cleanup transaction: 195 ms of the 268 ms average, with ~5,800 shared buffer hits per call even though it already uses relay_connection_bases_active_deadline. That index spans all 6.65M rows, and only a few hundred are ever active. Adds a partial index on the live rows alone, and reaps settled rows from relay_connection_bases and relay_direct_authorizations once their deadline is more than a day past. Both readers of either table require the row active/unconsumed and inside its deadline, and every deadline is set at most 30s past insert, so a settled row can never authorize anything again. The composite index stays: it is the only one covering active = 0, and it is what lets the drained reaper learn there is nothing to do from the index rather than the 1.5 GB heap. Measured at 200k rows, 5 buffers with it and 1,274 without. * test(relay): accept either bases index in the sweep plan assertion The negative assertion pinned a planner choice rather than the invariant: either index keeps the sweep off the 1.5 GB heap, and which one wins on cost is not something the test should fix. Matches how the same file already handles the two invite sweep indexes. Also names the column the authorization reaper actually measures, which is consumed_at rather than deadline. |
||
|
Jinwoo Hong
|
8e8a9b38ea |
perf(relay): stop indexing the column every control renewal writes (#21286)
* perf(relay): stop indexing the column every control renewal writes relay_assignment_activity_expiry indexes expires_at on relay_assignment_activity_leases, and expires_at is what every control renewal updates: ~471 calls/s, all of them non-HOT because a changed indexed column forbids HOT. The index has one reader, the 30s expiry sweep, which seq-scans the whole 14.8k-row table in under a millisecond. Drop it, and set fillfactor to 70 so a renewal has room for a second row version on its own page. Measured on postgres:16-alpine over 14.8k rows, WAL bytes per renewal and HOT ratio: index, fillfactor 100 (today) 0% HOT 371 B index, fillfactor 70 0% HOT 246 B no index, fillfactor 100 0.5% HOT 298 B no index, fillfactor 70 100% HOT 80 B Both are needed: the index makes HOT illegal, and the default fillfactor leaves no page space to make it possible. Neither statement can use the catalog pre-check as it stood. DROP INDEX IF EXISTS resolves the name before it locks, so once the index is gone it costs a catalog miss and takes no lock on the table - pinned in the lock-target census as the one exempt statement. ALTER TABLE SET does take a lock, so it gets a new 'reloption' target kind that asks pg_class.reloptions for the name=value pair, keeping the invariant that no lock-taking statement reaches a warm boot unchecked. * fix(relay): pre-check the activity-expiry drop and let it defer on a lock timeout The drop had no catalog pre-check, so it was sent on every boot, and a 55P03 from it was fatal: apply-postgres-schema throws on a lock timeout with no retry. On the migration boot that combination is a crash loop. All 28 directors reach the same DROP INDEX at once, it needs ACCESS EXCLUSIVE on a table written ~475/s with lock_timeout at 1s, and a boot that fails restarts the instance to re-queue the same DDL behind the same writers. Two changes: - A new 'index-by-name' lock target. A DROP INDEX names no table, so the existing index check could not serve it; this one resolves by name through the search_path with relkind = 'i', which is how the DROP itself resolves, and skips when absent. DROP INDEX now counts as lock-taking in the census, so it is covered rather than exempt, and IF EXISTS is required the way it is on DROP CONSTRAINT. - A 'schema-deferrable' marker, read from a statement's leading comment. A 55P03 on a marked statement logs orca_relay_postgres_schema_object_deferred and leaves the statement unapplied instead of failing the boot; the next boot re-sends it. Both activity-lease migrations carry it. Everything else keeps the old contract and still fails loudly. SchemaApplySummary gains a deferred count so a boot that skipped work is distinguishable from one with nothing to do. Verified against a real server: with the index present and the table held in ACCESS EXCLUSIVE by another session, both statements defer, the boot completes, nothing is half-applied, and the next boot finishes the job. A warm boot now sends neither statement at all. |
||
|
Jinwoo Hong
|
a3046cd27b |
fix(relay): treat database pool connect failures as transient, not director faults (#21243)
* fix(relay): treat pool connect failures as transient, not director faults pg-pool raises connection-acquire failures as a plain Error with no SQLSTATE, so the transient classifier matched only one of the three messages it can produce. The other two reached the routes unclassified and became HTTP 500s, which is what the rollout safety gate counts. The acquire boundary now marks the errors it produces, so "Connection terminated unexpectedly" counts as transient when the socket died during the handshake and stays a hard failure mid-statement, where a retry could repeat a commit whose outcome is unknown. /v1/regions and /v1/admin/evacuation-status gain the transient handling /v1/assign and /v1/resolve already had. * fix(relay): mirror the pool-connect verdict in failure diagnostics The query-failure event's connectionTimeout boolean matched one of the two messages connectionTimeoutMillis can produce, so the 210 dialling timeouts in the last day logged as false and were invisible to the field meant to find them. The pool-connect vocabulary now lives beside the acquire boundary that owns it, and both the router's classifier and the diagnostics read it from there, so the two cannot drift. The event also carries the routing verdict the caller already computed, making "how much of this burst reached users as a 500" one field. * fix(relay): null-safe transient classification and honest transient docs The classifier now runs inside the query catch, where a thrown null or undefined would have turned a database failure into a TypeError that buried it. The diagnostics doc claimed transient maps to a 503 or a 500. Sweeps, startup reconciliation, and admin routes that answer 409 all emit the same event, so counting the false ones over-states user-facing hard failures. |
||
|
Jinwoo Hong
|
0699d73fd6 |
fix(relay): skip boot-time DDL when the catalog already has the object (#21147)
* fix(relay): skip boot-time DDL when the catalog already has the object CREATE INDEX IF NOT EXISTS and ALTER TABLE ADD COLUMN IF NOT EXISTS take their relation lock before the server evaluates the existence test, so a boot on an already-migrated database still joins the lock queue. Relation locks are granted in queue order, so every writer queues behind it. The shared runner now asks pg_catalog whether the index or column is already there and skips the statement when a row comes back, and 55P03 is no longer retried by default: with the pre-check ahead of it, a lock timeout means the object is genuinely missing and each retry re-enters the queue. Push keeps the old retry behind an explicit option. * fix(relay): tie the index pre-check to its table and fail on an unreadable target Three defects found in review of the auth reference implementation: - The catalog query matched an index by name inside the table's namespace without checking it belonged to that table. Index names are unique per schema, not per table, so a same-named index on a sibling table answered yes and the real index was skipped forever. Added i.indrelid = t.oid. - Lock-target derivation read a keyword sitting in an identifier position as the object name: CREATE UNIQUE INDEX CONCURRENTLY ON t(c) yielded the name CONCURRENTLY, and ADD COLUMN IF NOT EXISTS with no column yielded IF. A wrong target is worse than none, so keywords are now excluded and an index or column statement whose target cannot be read throws at boot with the statement text instead of falling through to the lock path. - A concurrent-create collision retried the CREATE INDEX, taking SHARE on the table again for an object another director had just finished creating. The catalog is re-asked instead and a present object counts as skipped. * fix(relay): pre-check constraint swaps so a warm boot sends no DDL at all The two ALTER TABLE constraint statements were the last lock-taking statements without a pre-check, so every boot still took ACCESS EXCLUSIVE on relay_region_rehome_attempts twice. A lock target now carries the catalog answer that means there is nothing left to do. ADD CONSTRAINT skips when pg_constraint already names it; DROP CONSTRAINT IF EXISTS is the inverse and skips when it does not, because nothing to drop is nothing to do. The match is by name only: the CHECK body is generated from RELAY_REGIONS, so comparing it would re-run the swap on every region change. Changing a definition under the same name is an operator migration, and the rule comment beside SCHEMA says so. A bare DROP CONSTRAINT gets no target and throws at boot, because skipping it would swallow the undefined_object the server is supposed to raise. The census invariant is now that every lock-taking statement has a pre-check, with no exceptions, and the warm-boot Postgres test asserts zero statements sent rather than two. * fix(relay): refuse a multi-action ALTER TABLE instead of pre-checking its first action `ALTER TABLE t ADD COLUMN IF NOT EXISTS a TEXT, ADD COLUMN IF NOT EXISTS b TEXT` derived the target for `a` alone, so once `a` existed the whole statement was skipped and `b` was never added. The first subcommand parses, so neither the parse throw nor the census caught it. A lock-taking ALTER TABLE with a comma outside parentheses, quotes and comments now throws at boot. One action per statement, or no pre-check is possible. Commas inside a parenthesised type, a CHECK body, a quoted default or a comment are unaffected, and push's 18 statements still parse. * fix(relay): strip every comment before classifying, fold catalog names, count brackets Four findings from the bot reviews on #21147: - A comment between two keywords (ALTER TABLE t ADD /* note */ COLUMN c TEXT) was invisible to both the classification regexes and the must-parse shapes, so the statement got no target AND no throw and ran with no pre-check. Every comment is now stripped quote-aware before classification, nested block comments included. The server is still sent the original text. - hasTopLevelComma counted parentheses but not square brackets, so ADD COLUMN c bigint[] DEFAULT ARRAY[1, 2] read as two subcommands and failed the boot. - bareIdentifier split a qualified name on '.' regardless of quoting, so "a.b" became b", and it kept the written case while Postgres folds an unquoted identifier to lower case before storing it in relname, attname and conname. The name is now tokenised quote-aware and folded, with the qualified table text still passed to to_regclass as written. - sqlWithoutLeadingComments is renamed sqlWithoutComments to match. Relay's 74 statements and push's 18 all still parse, and no relay target name changed: every identifier there was already lower case. * fix(relay): treat a dollar-quoted body as opaque in both scanners A comment marker, comma, parenthesis or bracket inside `$$...$$` or `$tag$...$tag$` is text. The closing delimiter has to match the opening tag exactly, so an inner `$$` inside a `$tag$` body is more text rather than the end, and a tag cannot start with a digit, which keeps a `$1` placeholder from reading as an opener. Relay's pg_stat_statements DO block is the only dollar-quoted statement in the schema, and it now survives the stripper byte-identical. A test asserts that against the real statement. |
||
|
Jinwoo Hong
|
71e308e574 |
feat(relay): count failed cell-inventory lock acquisitions (#21067)
* feat(relay): count failed cell-inventory lock acquisitions The cell inventory lock is taken NOWAIT, so contention errors with 55P03 and retries instead of waiting. CellInventoryHoldSamples.record only runs after a successful acquisition, so the hold metrics were structurally blind to the dominant failure mode: production showed ~65 failed fleet-wide acquisitions per minute while cellInventoryHoldMsMax read a benign 53ms mean. Count failures next to the holds and publish them as cellInventoryLockUnavailable in orca_relay_runtime_metrics. Drained on both the commit and the rollback path, since a 55P03 rolls its transaction back. * fix(relay): separate request-path lock timeouts from sweep deferrals Review caught that the first counter only incremented under failIfUnavailable, which is the sweep mode. Background sweeps take the inventory NOWAIT and re-derive a skipped candidate next tick, so those deferrals are by design and already reported as orca_relay_sweep_cell_inventory_busy. The request path uses a bounded lock_timeout instead, whose expiry raises the same 55P03 without NOWAIT and was not counted at all -- so the metric measured only the benign population and missed the user-visible one. Split them: cellInventoryLockUnavailable for NOWAIT deferrals, cellInventoryLockTimeouts for expired bounded waits. Production over 30 minutes shows why the distinction matters -- roughly 1,200 fleet-wide sweep deferrals against roughly 10/min request-path timeouts. Adds transaction-path coverage for both drains, which were previously unpinned. Timeouts count per attempt, not per request, since 55P03 is retryable. * fix(relay): publish the cell-inventory lock metrics to Cloud Monitoring google_logging_metric.relay_snapshot only creates metrics for fields listed in relay_runtime_metrics, and the cellInventoryHold* fields were never added when the hold telemetry landed. They have been log-only since, so nothing could alert on the lock and the contention stayed invisible in exactly the way the telemetry was meant to prevent. Maps the three hold fields and both new failure counters. Also corrects the field comment: the split is by wait policy, not by caller. assignOnce takes the inventory fail-fast on its first placement attempt, so request-reachable sites land in cellInventoryLockUnavailable too; that lane reads as contention pressure, and the expired bounded wait is the stall lane. |
||
|
Jinwoo Hong
|
b07c4032ea | Log bounded PostgreSQL acquisition and execution failure diagnostics (#20749) | ||
|
Jinwoo Hong
|
d51747e4c4 | feat(relay): expose preloaded PostgreSQL statement statistics (#20712) | ||
|
Jinwoo Hong
|
cd9aa43a2c |
feat(relay): correct regional placement only when the source is idle (#20105)
* feat(relay): correct regional placement only at an idle source * test(relay): lock source activity capacity semantics |
||
|
Jinwoo Hong
|
f5be177e44 |
fix(relay): rehome hosts to their preferred region in either direction (#19241)
* fix(relay): rehome hosts to their preferred region in either direction The regional-rehome worker only moved hosts from a us-central1 cell to an asia-east2 one, so a host whose desktop later records us-central1 stays where it was put. Rehoming now compares the fresh preference against the region of the cell the host is on and moves it to a general cell in the preferred region either way, through the same drain, migrate, safety, and rate-limit machinery. - relay_region_rehome_attempts.preferred_region accepts both regions; existing databases are upgraded in place by an idempotent named-constraint swap that is safe when several directors start at once. - A target must carry the drain protocol too: moving a host onto a cell it can never be drained off again is the trap this change exists to undo. The fleet whose health gates a rehome is now every general drainable cell, which is exactly the set of legal sources and targets. - The trust probe accepts a source cell in any region. No wire change, and no behaviour change while the durable control is off. * fix(relay): bound bidirectional rehoming with a per-host cooldown Moving hosts in both directions removed the property that made the old one-way worker self-terminating: a desktop whose region probe flips would be dragged back and forth, one full drain and migrate per flip, because the preference age never expires while the host keeps reconnecting. - relay_region_rehome_control gains host_cooldown_ms, an operator input plumbed like preference_max_age_ms (workflow, ops script, admin route, durable row) and defaulted to seven days. A host with any attempt row inside the window, whichever way that move went, is not a candidate; the claim re-reads it under lock so an attempt landing between scan and claim cannot start a second move. Skips are named host_cooldown, and the lookup rides a new index on (user_id, relay_host_id, created_at). - The candidate scan now also requires the target cell to be enabled, so it mirrors the claim-time filter exactly and stops spending batch slots on candidates that are certain to be skipped. - Region CHECK lists are rendered from the shared region list instead of being written out four times. - The operations runbook states that cells without the drain protocol are neither sources, targets, nor members of the safety gate. * fix(relay): keep rehome reads and brakes working across the cooldown rollout The ops script validated hostCooldownMs on every inspected control, so against any director image predating the field inspect, pause, disable, and failed-enable recovery all threw client-side. The workflow always runs from main while the director image is operator-supplied, so that window opened at merge and reopened on every rollback: the operator lost read-only visibility and both emergency brakes while the worker could still be enabled. The field is now validated only when the director reports it, and every apply body that echoes an inspected control omits the key when that control lacks it, so a legacy director never sees an unknown key. The write path stays fail-closed the other way: enable refuses up front, before any mutation, when the director does not report a cooldown it could honour. Also replaces two bare 'us-central1' defaults with RELAY_DEFAULT_REGION. |
||
|
Jinwoo Hong
|
b6f453df06 |
perf(relay): per-cell inventory locks, delta counters, and a pool statement timeout (#18722)
The sticky refresh path and reservation reconciliation both took the fleet-wide `relay_cells ... FOR UPDATE` scan to mutate one or two rows, so one busy cell queued unrelated reconnects and migration completions behind it. Both now lock only the rows they touch, in the same ascending cell_id order, and the sticky grant moves its counter by a delta instead of writing back a snapshot value. Placement keeps the ordered inventory lock: choosing the least-loaded cell is a genuinely fleet-wide decision, and dynamically locking only the selected target is what allowed cross-cell cycles before. The pool's statement_timeout becomes env-configurable and a 57014 now reaches the bounded transaction retry instead of surfacing as a terminal failure. Schema DDL moves to its own `max: 1`, statement_timeout-free pool that is ended before the serving pool opens, so a slow CREATE INDEX cannot inherit a request deadline it will never fit inside. |
||
|
Jinwoo Hong
|
d66386bc82 |
fix(cloud): bound and yield the relay's global cell-inventory lock (#18521)
Mirrors stablyai/orca-cloud#471 (squash c3354e8), byte-identical under cloud/. The relay's cell-inventory lock (SELECT ... FROM relay_cells FOR UPDATE over all 23 rows) is one global critical section shared by the assignment hot path and every director sweep; with the pool's 1s lock_timeout a blocked waiter held a pooled client for a full second, producing ~690 55P03 retries per 5 minutes in production. Request paths now bound the wait at 500ms with a SET LOCAL that is restored to the pool default before the next statement; director-only sweeps take the lock NOWAIT and skip the tick; sweep timers are jittered; hold time is exported as additive runtime-metrics fields so the bound can be tuned. |
||
|
Jinwoo Hong
|
3eec77c11a |
chore(cloud): add the relay fence broker, ops console, Terraform root, scripts, and 24 cloud-* workflows (#18413)
Phase 6 of the relay split: the relay's deploy/operate surface moves under cloud/ with 24 cloud-* workflows gated on ORCA_CLOUD_OPERATIONS_ENABLED, the Cloud SQL rollout lease action, the relay Terraform root (dual-accept identities for both repositories), scripts, docs, CODEOWNERS, and a terraform validate job in Cloud Verify. |