Commit Graph
11 Commits
Author SHA1 Message Date
OrcaWinandOrcaWin 7d6603aa17 [comment audit] Complete repository-wide comment cleanup (#33249)
* [comment audit] Align legacy failure-read counts with cached display

* [comment audit] Condense shared provider lifecycle and browser recovery comments [skip ci]

* [comment audit] Condense shared generation, path and preview comments [skip ci]

* [comment audit] Condense host, file and Git contract rationale [skip ci]

* [comment audit] Tighten Git lifecycle and provider contract comments [skip ci]

* [comment audit] Clarify shared host authority and launch isolation [skip ci]

* [comment audit] simplify local access mobile and native chat contracts [skip ci]

* [comment audit] simplify native chat presentation and state contracts [skip ci]

* [comment audit] simplify native runtime and provider contracts [skip ci]

* [comment audit] simplify migration and orchestration contracts [skip ci]

* [comment audit] simplify plugin trust and process capture rationale [skip ci]

* [comment audit] simplify terminal delivery and remote compatibility rationale [skip ci]

* [comment audit] clarify remote recovery and launch wire rationale [skip ci]

* [comment audit] shorten shared RPC compatibility and runtime evidence comments [skip ci]

* [comment audit] simplify secure storage shell startup and sidebar rationale [skip ci]

* [comment audit] clarify sidebar, skill and SSH invariants [skip ci]

* [comment audit] condense structured chat delivery and timing rationale [skip ci]

* [comment audit] condense terminal replay and telemetry invariants [skip ci]

* [comment audit] clarify launch permission and terminal delivery rationale [skip ci]

* [comment audit] clarify workspace ownership and layout contracts [skip ci]

* [comment audit] wait for fitted and painted SSH terminal links

* [comment audit] clarify layout persistence and session ownership [skip ci]

* [comment audit] clarify host routing and workspace lifecycle contracts [skip ci]

* [comment audit] clarify startup persistence and app lifetime comments [skip ci]

* [comment audit] clarify recovery and activity boundaries [skip ci]

* [comment audit] condense activity and artifact invariants [skip ci]

* [comment audit] condense automation authority contracts [skip ci]

* [comment audit] simplify automation scheduling and history contracts [skip ci]

* [comment audit] clarify browser chrome and annotation lifecycles [skip ci]

* [comment audit] clarify browser routing and retained guest ownership [skip ci]

* [comment audit] clarify browser recovery and document ownership comments [skip ci]

* [comment audit] restore hosted runners required by native IME fixtures

* [comment audit] shorten preview, palette and dashboard invariants [skip ci]

* [comment audit] clarify terminal preview, dashboard and retention comments [skip ci]

* [comment audit] clarify diff-note recovery and editor lifecycle comments [skip ci]

* [comment audit] shorten rich-text and combined-diff ownership comments [skip ci]

* [comment audit] shorten autosave, draft recovery and model ownership comments [skip ci]

* [comment audit] shorten save-queue, notebook and Markdown constraints [skip ci]

* [comment audit] shorten Monaco, PDF and rich Markdown safeguards [skip ci]

* [comment audit] clarify rich Markdown source and selection safeguards [skip ci]

* [comment audit] clarify editor loading and emulator stream lifetimes [skip ci]

* [comment audit] trim emulator and feature tour narration [skip ci]

* [comment audit] clarify floating workspace and GitHub cache ownership [skip ci]

* [comment audit] trim provider controls and Linear filter explanations [skip ci]

* [comment audit] clarify pairing and chat resume lifetimes [skip ci]

* [comment audit] clarify chat delivery and transcript lifetimes [skip ci]

* [comment audit] clarify session options and host-owned status [skip ci]

* [comment audit] clarify draft durability and live-status precedence [skip ci]

* [comment audit] Trim chat send and model ownership explanations [skip ci]

* [comment audit] Shorten transcript ownership and send recovery comments [skip ci]

* [comment audit] Simplify native chat composer and transcript hook comments [skip ci]

* [comment audit] Shorten native chat scroll and structured session explanations [skip ci]

* [comment audit] Trim workspace picker and onboarding commentary [skip ci]

* [comment audit] Simplify pet rendering, review cache and quick open comments [skip ci]

* [comment audit] Trim sidebar and session history explanations [skip ci]

* [comment audit] Condense session resume and review checks comments [skip ci]

* [comment audit] Shorten checks state and file explorer refresh comments [skip ci]

* [comment audit] Trim plugin bridge and source-control action rationale [skip ci]

* [comment audit] Trim source-control listing and review rationale [skip ci]

* [comment audit] Shorten sidebar lifecycle and explorer safeguards [skip ci]

* [comment audit] Condense Settings account and appearance rationale [skip ci]

* [comment audit] Trim pairing and repository Settings explanations [skip ci]

* [comment audit] Condense Settings search SSH and terminal safeguards [skip ci]

* [comment audit] Shorten Settings voice and provider ownership comments [skip ci]

* [comment audit] Condense Settings draft persistence and host-state rationale [skip ci]

* [comment audit] Trim Settings hooks and sidebar setup rationale [skip ci]

* [comment audit] Condense sidebar deletion and board-filter safeguards [skip ci]

* [comment audit] Condense worktree cards, host ownership and Markdown contracts [skip ci]

* [comment audit] Condense sidebar drag, folder and filtering contracts [skip ci]

* [comment audit] Condense sidebar parity, sleep and host ownership contracts [skip ci]

* [comment audit] Prune focus-read baseline after comment cleanup

* [comment audit] Clarify sidebar visibility, board drops and agent evidence [skip ci]

* [comment audit] Tighten sidebar listing, drag and reveal contracts [skip ci]

* [comment audit] Clarify viewport geometry and skill update safeguards [skip ci]

* [comment audit] Tighten skill inventory, usage and host status comments [skip ci]

* [comment audit] Clarify usage ownership and resource kill safeguards [skip ci]

* [comment audit] Tighten tab selection, rename and search contracts [skip ci]

* [comment audit] Tighten tab-strip geometry and drag ownership comments [skip ci]

* [comment audit] Tighten Tasks recovery, pagination and mutation comments [skip ci]

* [comment audit] Tighten provider loading and terminal recovery comments [skip ci]

* [comment audit] Tighten completion, cadence and PTY ownership comments [skip ci]

* [comment audit] Clarify terminal restore and parked-pane boundaries [skip ci]

* [comment audit] Tighten PTY startup and replay explanations [skip ci]

* [comment audit] Clarify PTY identity and delivery ownership [skip ci]

* [comment audit] Shorten remote reconnect and input replay rationale [skip ci]

* [comment audit] Condense terminal parking clipboard and file link comments [skip ci]

* [comment audit] Clarify IME ownership and trim trace history [skip ci]

* [comment audit] Shorten pane recovery and parked watcher explanations [skip ci]

* [comment audit] Condense snapshot replay and visibility recovery rationale [skip ci]

* [comment audit] Shorten terminal parking and keyboard ownership rationale [skip ci]

* [comment audit] Trim renderer lifecycle and menu rationale [skip ci]

* [comment audit] Condense workspace cleanup ownership and settlement notes [skip ci]

* [comment audit] Clarify renderer hook evidence and source ownership [skip ci]

* [comment audit] Tighten IPC ownership and reconnect explanations [skip ci]

* [comment audit] Trim renderer focus, paste and attention comments [skip ci]

* [comment audit] Shorten scroll, locale and agent lifecycle comments [skip ci]

* [comment audit] Trim agent launch, resume and ownership commentary [skip ci]

* [comment audit] Shorten account, browser and editor comments [skip ci]

* [comment audit] Clarify host-mirror waits, previews and input ownership [skip ci]

* [comment audit] Condense launch recovery and editor tokenization comments [skip ci]

* [comment audit] Condense workspace selection and terminal pane rationale [skip ci]

* [comment audit] Condense terminal fitting and rendering recovery comments [skip ci]

* [comment audit] Condense terminal input and host routing explanations [skip ci]

* [comment audit] Condense ownership and sleeping-session recovery rationale [skip ci]

* [comment audit] Condense structured launch and terminal link explanations [skip ci]

* [comment audit] Condense terminal ownership and workspace file routing rationale [skip ci]

* [comment audit] Condense session persistence and workspace activation rationale [skip ci]

* [comment audit] Condense workspace routing and browser ownership rationale [skip ci]

* [comment audit] Condense mirror recovery and layout authority rationale [skip ci]

* [comment audit] Condense runtime transport and file safety rationale [skip ci]

* [comment audit] Condense runtime graph and structured session rationale [skip ci]

* [comment audit] Condense browser creation and remote tab recovery rationale [skip ci]

* [comment audit] Condense snapshot authority and publisher recovery rationale [skip ci]

* [comment audit] Condense frame, startup and provider cache rationale [skip ci]

* [comment audit] Condense host twin and agent lifecycle rationale [skip ci]

* [comment audit] Condense browser authority and editor cleanup rationale [skip ci]

* [comment audit] Condense editor buffer identity and review cache rationale [skip ci]

* [comment audit] Condense provider lifetime and repository ownership rationale [skip ci]

* [comment audit] Condense reconnect, hydration and terminal retirement rationale [skip ci]

* [comment audit] Condense recovery, reconciliation and persistence rationale [skip ci]

* [comment audit] Condense cleanup ownership and listing admission rationale [skip ci]

* [comment audit] Condense teardown ordering and terminal ownership rationale [skip ci]

* [comment audit] Condense recovery ledger and web host-boundary rationale [skip ci]

* [comment audit] Condense web reconnect and credential lifecycle rationale [skip ci]

* [comment audit] Condense cloud rollout and assignment safety rationale [skip ci]

* [comment audit] Condense relay session and database recovery rationale [skip ci]

* [comment audit] Condense relay placement and protocol compatibility rationale [skip ci]

* [comment audit] Condense build and packaging constraints [skip ci]

* [comment audit] Condense CI gates and validation isolation rationale [skip ci]

* [comment audit] Condense catalog identity and browser validation rationale [skip ci]

* [comment audit] Condense mobile browser and frame-budget rationale [skip ci]

* [comment audit] Shorten mobile route and browser test explanations [skip ci]

* [comment audit] Condense editor and session seam test rationale [skip ci]

* [comment audit] Trim session and terminal closure history [skip ci]

* [comment audit] Shorten terminal browser and census explanations [skip ci]

* [comment audit] Preserve root-only E2E diagnostic baseline

* [comment audit] Condense mobile seam and HTML preview test rationale [skip ci]

* [comment audit] Condense preview instrumentation and render harness rationale [skip ci]

* [comment audit] Condense bundle contracts and native-build constraints [skip ci]

* [comment audit] Shorten native packaging and CI routing rationale [skip ci]

* [comment audit] Condense release tooling and benchmark constraints [skip ci]

* [comment audit] Condense packaging and platform gate rationale [skip ci]

* [comment audit] Shorten mobile route and history rationale [skip ci]

* [comment audit] Condense mobile browser and component rationale [skip ci]

* [comment audit] Shorten drawer and workspace-form rationale [skip ci]

* [comment audit] Trim mobile review and editor-document explanations [skip ci]

* [comment audit] Condense editor lifecycle and Markdown explanations [skip ci]

* [comment audit] Trim mobile file, diagnostics and dictation commentary [skip ci]

* [comment audit] Simplify mobile host-list and workspace-view explanations [skip ci]

* [comment audit] Clarify mobile shell lifetime and bridge delivery constraints [skip ci]

* [comment audit] Condense bridge stream, native request and audio rationale [skip ci]

* [comment audit] Reduce bridge schema and page-client commentary [skip ci]

* [comment audit] Tighten mobile page bridge contracts [skip ci]

* [comment audit] Clarify mobile cache and session constraints [skip ci]

* [comment audit] Condense mobile route and lifecycle rationale [skip ci]

* [comment audit] Tighten mobile shell hook timing rationale [skip ci]

* [comment audit] Clarify route handoff and notification compatibility [skip ci]

* [comment audit] Clarify push registration and dictation cleanup constraints [skip ci]

* [comment audit] Condense keyboard geometry and native media ownership rationale [skip ci]

* [comment audit] Tighten mobile chat scroll and composer concurrency rationale [skip ci]

* [comment audit] Condense mobile reply compatibility and upload ownership rationale [skip ci]

* [comment audit] Condense optimistic echo and scroll ownership rationale [skip ci]

* [comment audit] Tighten stream replay and queued session ownership rationale [skip ci]

* [comment audit] Condense mobile composer and reply-schema constraints [skip ci]

* [comment audit] Clarify mobile transcript recovery and session ownership [skip ci]

* [comment audit] Clarify mobile queue and source-control compatibility [skip ci]

* [comment audit] Condense storage and task launch constraints [skip ci]

* [comment audit] Clarify task schemas and terminal document ownership [skip ci]

* [comment audit] Tighten terminal input and viewport rationale [skip ci]

* [comment audit] Shorten terminal and RPC recorder contracts [skip ci]

* [comment audit] Condense recorder engine and connection contracts [skip ci]

* [comment audit] Shorten host credential and relay recovery comments [skip ci]

* [comment audit] Tighten pairing recovery and bundle transport rationale [skip ci]

* [comment audit] Clarify RPC delivery and workspace catalog boundaries [skip ci]

* [comment audit] Condense catalog and relay entry contracts [skip ci]

* [comment audit] Shorten shutdown and SSH relay delivery contracts [skip ci]

* [comment audit] Condense WSL relay and E2E regression rationale [skip ci]

* [comment audit] Shorten cross-version harness and renderer fixture explanations [skip ci]

* [comment audit] Condense paired-host and E2E lifecycle rationale [skip ci]

* [comment audit] Shorten layout oracle and regression explanations [skip ci]

* [comment audit] Condense paired recovery and ownership regression contracts [skip ci]

* [comment audit] Shorten restart, sidebar and SSH lifecycle contracts [skip ci]

* [comment audit] Condense terminal rendering and IME regression rationale [skip ci]

* [comment audit] Shorten terminal recovery and benchmark contracts [skip ci]

* [comment audit] Condense tooling and Windows lifecycle proof rationale [skip ci]

* [comment audit] Condense workflow routing and release rationale [skip ci]

* [comment audit] Condense mobile security and installer contracts [skip ci]

* [comment audit] Shorten native rationale and stylesheet workarounds [skip ci]

* [comment audit] Shorten Windows visibility probe rationale [skip ci]

* [comment audit] Condense embedded release and signing rationale [skip ci]

* [comment audit] Retain current main test behavior in final reconciliation [skip ci]

* [comment audit] Complete final comment review across current source inventory

* [comment audit] Handle consolidated audit reports before final CI

* [comment audit] Preserve credential storage census detail [skip ci]

* [comment audit] Scale broad changed E2E selections with verified timing shards

* [comment audit] Align E2E assertions with browser close and settled session metadata

* [comment audit] Match existing E2Es to current resource and windowing contracts

* [comment audit] Observe the shared server outage notice in reconnect coverage

* [comment audit] Make existing renderer probes measure settled tab and PTY work

* [comment audit] Align palette E2E selector with task search

* [comment audit] Refresh remaining palette search selectors

* [comment audit] Keep selector correction within selected audit specs

* [comment audit] Await emoji fixture settings inside renderer evaluation

* [comment audit] Cancel fixture fit retries between terminal tests

---------

Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com>
2026-10-25 04:05:46 -07:00
OrcaWinandOrcaWin 8f6283b32b test: exercise real boundaries and trim redundant cases (#32353)
* test: replace redundant checks with observed behavior

* docs: reconcile completed audit batches and upstream review

---------

Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com>
2026-10-23 04:32:31 -07:00
Jinwoo Hong e72044a936 fix(relay): a lost database reply ends its connection within statement timeout + 10 s (#26829)
* fix(relay): a lost database reply ends its connection within statement timeout + 2 s

The request pool sets query_timeout to the statement timeout plus 2 s. A
read timeout poisons its transaction: later statements fail at once,
COMMIT never runs (even when the operation caught the error), no ROLLBACK
queues behind the dead statement, and the client is released with the
error so pg-pool destroys it. Autocommit statements destroy their client
too. The server bounds an orphaned idle backend with idle_session_timeout
30 s. A read timeout is answered 503 and never retried, and a renewal
batch that loses its reply fails whole instead of fanning out.

* fix(relay): read timeout margin 10 s, configurable, so routine 5-7 s database stalls never trip it

A whole-VM stall delays replies the server already timed. With a 2 s
margin the read timeout fired inside routine stalls, destroying healthy
clients and failing requests that succeed today. The margin is now 10 s
(ORCA_RELAY_POSTGRES_READ_TIMEOUT_MARGIN_MS), and a real-server test holds
a reply 6.5 s and requires it delivered on the same connection. The fix
level rises to 2 here, with the lost-reply fix.

* docs(relay): name the startup retry that can see a read timeout, and why it is safe

Boot open retries only an unreachable database. The director's startup
cell reconcile does retry a lost reply, which is safe: it re-runs upserts
and limit-change deletes that converge on the same rows.

* fix(relay): answer 503, not 500, when Postgres ends the session mid-request

Staging B1: pg_terminate_backend during /v1/assign returned 500 29 times.
57P01/57P02 and 08000/08003 join the transient SQLSTATEs; pg's
'Connection terminated' and ECONNRESET count only when the database layer
threw them. transaction() still never retries them.

* fix(relay): answer 503 when the pool acquire fails on a dead socket

Staging B1 re-test: a Cloud SQL restart failed PostgresDatabase.transaction's
acquire with 'write EPIPE', answered 500. An acquire failure with EPIPE,
ECONNRESET, ECONNREFUSED, ETIMEDOUT or EHOSTUNREACH is now a pool connect
failure (no statement ran), and acquire failures count as database-layer
errors. EPIPE mid-statement is treated like ECONNRESET.
2026-10-09 18:00:10 -04:00
Jinwoo Hong eb6068a434 fix(relay): stop a terminated checked-out PostgreSQL client from killing the cell (#21840)
pg-pool removes its own `error` listener when it hands a client out
(pg-pool@3.14.0 index.js:344) and only reattaches it in `_release`
(index.js:385). Between acquire and release the client therefore has no
`error` listener, so when Cloud SQL terminates that session mid-statement
the emit becomes an unhandled 'error' event and the process exits.
`absorbPostgresIdleClientErrors` cannot see it: pg-pool routes to
`pool.on('error')` only from the idle listener.

Attach a per-checkout `error` listener in the one seam every relay
checkout passes through, log a single warn line, and release the client
with the error so pg-pool destroys it instead of pooling a dead
connection. The listener is removed on release so it cannot accumulate.
The in-flight query still rejects, so existing failure reporting and the
transaction retry ladder are unchanged.

Claude-Session: https://claude.ai/session/ced32ebb-7155-4413-adad-1eccd14c2010
2026-09-20 17:46:25 -04:00
Jinwoo Hong 8e8a9b38ea perf(relay): stop indexing the column every control renewal writes (#21286)
* perf(relay): stop indexing the column every control renewal writes

relay_assignment_activity_expiry indexes expires_at on
relay_assignment_activity_leases, and expires_at is what every control
renewal updates: ~471 calls/s, all of them non-HOT because a changed
indexed column forbids HOT. The index has one reader, the 30s expiry
sweep, which seq-scans the whole 14.8k-row table in under a millisecond.

Drop it, and set fillfactor to 70 so a renewal has room for a second
row version on its own page. Measured on postgres:16-alpine over 14.8k
rows, WAL bytes per renewal and HOT ratio:

  index, fillfactor 100 (today)     0% HOT   371 B
  index, fillfactor 70              0% HOT   246 B
  no index, fillfactor 100        0.5% HOT   298 B
  no index, fillfactor 70         100% HOT    80 B

Both are needed: the index makes HOT illegal, and the default fillfactor
leaves no page space to make it possible.

Neither statement can use the catalog pre-check as it stood. DROP INDEX
IF EXISTS resolves the name before it locks, so once the index is gone
it costs a catalog miss and takes no lock on the table - pinned in the
lock-target census as the one exempt statement. ALTER TABLE SET does
take a lock, so it gets a new 'reloption' target kind that asks
pg_class.reloptions for the name=value pair, keeping the invariant that
no lock-taking statement reaches a warm boot unchecked.

* fix(relay): pre-check the activity-expiry drop and let it defer on a lock timeout

The drop had no catalog pre-check, so it was sent on every boot, and a
55P03 from it was fatal: apply-postgres-schema throws on a lock timeout
with no retry. On the migration boot that combination is a crash loop.
All 28 directors reach the same DROP INDEX at once, it needs ACCESS
EXCLUSIVE on a table written ~475/s with lock_timeout at 1s, and a boot
that fails restarts the instance to re-queue the same DDL behind the
same writers.

Two changes:

- A new 'index-by-name' lock target. A DROP INDEX names no table, so the
  existing index check could not serve it; this one resolves by name
  through the search_path with relkind = 'i', which is how the DROP
  itself resolves, and skips when absent. DROP INDEX now counts as
  lock-taking in the census, so it is covered rather than exempt, and
  IF EXISTS is required the way it is on DROP CONSTRAINT.

- A 'schema-deferrable' marker, read from a statement's leading comment.
  A 55P03 on a marked statement logs
  orca_relay_postgres_schema_object_deferred and leaves the statement
  unapplied instead of failing the boot; the next boot re-sends it. Both
  activity-lease migrations carry it. Everything else keeps the old
  contract and still fails loudly. SchemaApplySummary gains a deferred
  count so a boot that skipped work is distinguishable from one with
  nothing to do.

Verified against a real server: with the index present and the table
held in ACCESS EXCLUSIVE by another session, both statements defer, the
boot completes, nothing is half-applied, and the next boot finishes the
job. A warm boot now sends neither statement at all.
2026-09-17 16:53:39 -04:00
Jinwoo Hong 0699d73fd6 fix(relay): skip boot-time DDL when the catalog already has the object (#21147)
* fix(relay): skip boot-time DDL when the catalog already has the object

CREATE INDEX IF NOT EXISTS and ALTER TABLE ADD COLUMN IF NOT EXISTS take
their relation lock before the server evaluates the existence test, so a
boot on an already-migrated database still joins the lock queue. Relation
locks are granted in queue order, so every writer queues behind it.

The shared runner now asks pg_catalog whether the index or column is
already there and skips the statement when a row comes back, and 55P03
is no longer retried by default: with the pre-check ahead of it, a lock
timeout means the object is genuinely missing and each retry re-enters
the queue. Push keeps the old retry behind an explicit option.

* fix(relay): tie the index pre-check to its table and fail on an unreadable target

Three defects found in review of the auth reference implementation:

- The catalog query matched an index by name inside the table's namespace
  without checking it belonged to that table. Index names are unique per
  schema, not per table, so a same-named index on a sibling table answered
  yes and the real index was skipped forever. Added i.indrelid = t.oid.
- Lock-target derivation read a keyword sitting in an identifier position as
  the object name: CREATE UNIQUE INDEX CONCURRENTLY ON t(c) yielded the name
  CONCURRENTLY, and ADD COLUMN IF NOT EXISTS with no column yielded IF. A
  wrong target is worse than none, so keywords are now excluded and an index
  or column statement whose target cannot be read throws at boot with the
  statement text instead of falling through to the lock path.
- A concurrent-create collision retried the CREATE INDEX, taking SHARE on the
  table again for an object another director had just finished creating. The
  catalog is re-asked instead and a present object counts as skipped.

* fix(relay): pre-check constraint swaps so a warm boot sends no DDL at all

The two ALTER TABLE constraint statements were the last lock-taking
statements without a pre-check, so every boot still took ACCESS EXCLUSIVE
on relay_region_rehome_attempts twice.

A lock target now carries the catalog answer that means there is nothing
left to do. ADD CONSTRAINT skips when pg_constraint already names it; DROP
CONSTRAINT IF EXISTS is the inverse and skips when it does not, because
nothing to drop is nothing to do. The match is by name only: the CHECK body
is generated from RELAY_REGIONS, so comparing it would re-run the swap on
every region change. Changing a definition under the same name is an
operator migration, and the rule comment beside SCHEMA says so.

A bare DROP CONSTRAINT gets no target and throws at boot, because skipping
it would swallow the undefined_object the server is supposed to raise.

The census invariant is now that every lock-taking statement has a
pre-check, with no exceptions, and the warm-boot Postgres test asserts zero
statements sent rather than two.

* fix(relay): refuse a multi-action ALTER TABLE instead of pre-checking its first action

`ALTER TABLE t ADD COLUMN IF NOT EXISTS a TEXT, ADD COLUMN IF NOT EXISTS b
TEXT` derived the target for `a` alone, so once `a` existed the whole
statement was skipped and `b` was never added. The first subcommand parses,
so neither the parse throw nor the census caught it.

A lock-taking ALTER TABLE with a comma outside parentheses, quotes and
comments now throws at boot. One action per statement, or no pre-check is
possible. Commas inside a parenthesised type, a CHECK body, a quoted
default or a comment are unaffected, and push's 18 statements still parse.

* fix(relay): strip every comment before classifying, fold catalog names, count brackets

Four findings from the bot reviews on #21147:

- A comment between two keywords (ALTER TABLE t ADD /* note */ COLUMN c
  TEXT) was invisible to both the classification regexes and the must-parse
  shapes, so the statement got no target AND no throw and ran with no
  pre-check. Every comment is now stripped quote-aware before classification,
  nested block comments included. The server is still sent the original text.
- hasTopLevelComma counted parentheses but not square brackets, so
  ADD COLUMN c bigint[] DEFAULT ARRAY[1, 2] read as two subcommands and
  failed the boot.
- bareIdentifier split a qualified name on '.' regardless of quoting, so
  "a.b" became b", and it kept the written case while Postgres folds an
  unquoted identifier to lower case before storing it in relname, attname
  and conname. The name is now tokenised quote-aware and folded, with the
  qualified table text still passed to to_regclass as written.
- sqlWithoutLeadingComments is renamed sqlWithoutComments to match.

Relay's 74 statements and push's 18 all still parse, and no relay target
name changed: every identifier there was already lower case.

* fix(relay): treat a dollar-quoted body as opaque in both scanners

A comment marker, comma, parenthesis or bracket inside `$$...$$` or
`$tag$...$tag$` is text. The closing delimiter has to match the opening tag
exactly, so an inner `$$` inside a `$tag$` body is more text rather than the
end, and a tag cannot start with a digit, which keeps a `$1` placeholder
from reading as an opener.

Relay's pg_stat_statements DO block is the only dollar-quoted statement in
the schema, and it now survives the stripper byte-identical. A test asserts
that against the real statement.
2026-09-17 01:00:28 -04:00
Jinwoo Hong d51747e4c4 feat(relay): expose preloaded PostgreSQL statement statistics (#20712) 2026-09-14 17:32:49 -04:00
Jinwoo Hong f5be177e44 fix(relay): rehome hosts to their preferred region in either direction (#19241)
* fix(relay): rehome hosts to their preferred region in either direction

The regional-rehome worker only moved hosts from a us-central1 cell to an
asia-east2 one, so a host whose desktop later records us-central1 stays where
it was put. Rehoming now compares the fresh preference against the region of
the cell the host is on and moves it to a general cell in the preferred
region either way, through the same drain, migrate, safety, and rate-limit
machinery.

- relay_region_rehome_attempts.preferred_region accepts both regions; existing
  databases are upgraded in place by an idempotent named-constraint swap that
  is safe when several directors start at once.
- A target must carry the drain protocol too: moving a host onto a cell it
  can never be drained off again is the trap this change exists to undo. The
  fleet whose health gates a rehome is now every general drainable cell,
  which is exactly the set of legal sources and targets.
- The trust probe accepts a source cell in any region.

No wire change, and no behaviour change while the durable control is off.

* fix(relay): bound bidirectional rehoming with a per-host cooldown

Moving hosts in both directions removed the property that made the old
one-way worker self-terminating: a desktop whose region probe flips would be
dragged back and forth, one full drain and migrate per flip, because the
preference age never expires while the host keeps reconnecting.

- relay_region_rehome_control gains host_cooldown_ms, an operator input
  plumbed like preference_max_age_ms (workflow, ops script, admin route,
  durable row) and defaulted to seven days. A host with any attempt row
  inside the window, whichever way that move went, is not a candidate; the
  claim re-reads it under lock so an attempt landing between scan and claim
  cannot start a second move. Skips are named host_cooldown, and the lookup
  rides a new index on (user_id, relay_host_id, created_at).
- The candidate scan now also requires the target cell to be enabled, so it
  mirrors the claim-time filter exactly and stops spending batch slots on
  candidates that are certain to be skipped.
- Region CHECK lists are rendered from the shared region list instead of
  being written out four times.
- The operations runbook states that cells without the drain protocol are
  neither sources, targets, nor members of the safety gate.

* fix(relay): keep rehome reads and brakes working across the cooldown rollout

The ops script validated hostCooldownMs on every inspected control, so
against any director image predating the field inspect, pause, disable, and
failed-enable recovery all threw client-side. The workflow always runs from
main while the director image is operator-supplied, so that window opened at
merge and reopened on every rollback: the operator lost read-only visibility
and both emergency brakes while the worker could still be enabled.

The field is now validated only when the director reports it, and every apply
body that echoes an inspected control omits the key when that control lacks
it, so a legacy director never sees an unknown key. The write path stays
fail-closed the other way: enable refuses up front, before any mutation, when
the director does not report a cooldown it could honour.

Also replaces two bare 'us-central1' defaults with RELAY_DEFAULT_REGION.
2026-09-07 04:40:37 -04:00
Jinwoo Hong b6f453df06 perf(relay): per-cell inventory locks, delta counters, and a pool statement timeout (#18722)
The sticky refresh path and reservation reconciliation both took the fleet-wide
`relay_cells ... FOR UPDATE` scan to mutate one or two rows, so one busy cell
queued unrelated reconnects and migration completions behind it. Both now lock
only the rows they touch, in the same ascending cell_id order, and the sticky
grant moves its counter by a delta instead of writing back a snapshot value.

Placement keeps the ordered inventory lock: choosing the least-loaded cell is a
genuinely fleet-wide decision, and dynamically locking only the selected target
is what allowed cross-cell cycles before.

The pool's statement_timeout becomes env-configurable and a 57014 now reaches
the bounded transaction retry instead of surfacing as a terminal failure.
Schema DDL moves to its own `max: 1`, statement_timeout-free pool that is ended
before the serving pool opens, so a slow CREATE INDEX cannot inherit a request
deadline it will never fit inside.
2026-09-04 18:18:46 -04:00
Jinwoo Hong dd9eaa9585 fix(cloud): retry the committed-winner collision codes in relay schema startup (#18553)
* fix(cloud): retry the committed-winner collision codes in relay schema startup

`CREATE TABLE IF NOT EXISTS` only checks the name before the catalog inserts, so
the loser of a concurrent CREATE fails in one of two ways depending on timing:
on the catalog unique index (23505, which the startup retry already handled) or,
when the winner has committed by the time the loser reaches TypeCreate /
heap_create_with_catalog, on the name check those routines repeat (42710
duplicate type, 42P07 duplicate relation). The predicate treated the latter as
fatal, so a director could fail startup on a table it was about to find present.

This is what turned `postgres-schema-concurrency-postgres.test.ts` red on main
and on every relay PR (CI's shared runner loses the race more often than a dev
box): a throwaway diagnostic run in CI reported 42710 from TypeCreate and 42P07
from heap_create_with_catalog as the only rejection reasons.

Treat 42710/42P07 as retryable for `CREATE TABLE IF NOT EXISTS` and 42P07 for
`CREATE [UNIQUE] INDEX IF NOT EXISTS`; every other statement shape still fails
fast. The concurrency test now runs ten rounds and reports the loser's SQLSTATE
instead of a bare boolean.

* chore(cloud): allowlist the RFC 6455 example Sec-WebSocket-Key for upgrade tests

Cloud Verify's Secret scan runs gitleaks over --all refs, so the raw-socket
upgrade test on fix/relay-upgrade-malformed-uri (#18547) trips every cloud PR's
scan until its allowlist reaches main. Land the allowlist here first.
2026-09-03 21:20:27 -04:00
Jinwoo Hong 3eec77c11a chore(cloud): add the relay fence broker, ops console, Terraform root, scripts, and 24 cloud-* workflows (#18413)
Phase 6 of the relay split: the relay's deploy/operate surface moves under cloud/ with 24 cloud-* workflows gated on ORCA_CLOUD_OPERATIONS_ENABLED, the Cloud SQL rollout lease action, the relay Terraform root (dual-accept identities for both repositories), scripts, docs, CODEOWNERS, and a terraform validate job in Cloud Verify.
2026-09-03 06:55:14 -04:00