Commit Graph
4 Commits
Author SHA1 Message Date
OrcaWinandOrcaWin 89b3d38284 test: audit delayed replies and database boundaries (#32426)
* test: audit delayed replies and database boundaries

* docs: reconcile the fourth test audit batch

---------

Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com>
2026-10-23 05:33:18 -07:00
Neil 39815a086f Reduce CI import, frame-capture, teardown, and shard costs (#28879)
* Avoid unused module imports and trim cosmetic test coverage

* Capture hidden popup frames and attach CDP before guest creation

* Overlap relay test projects within the existing worker budget

* Consolidate native preparation and add an opt-in transform cache comparison

* Release inherited debugger sockets during final E2E teardown

* Refresh unit scheduling weights from all five qualified ARM shards

* Avoid retaining unasserted RPC call history in the delivery model

* Use the supported unconditional background wait syntax

* Overlap unconditional output upload with native preparation

* Verify relay integration stays enabled for ordinary unit callers
2026-10-14 17:10:55 -07:00
Jinwoo Hong 9c41d9e8d8 feat(relay): per-cell switches for the rejection fence and the lost-reply margin (#27057)
* feat(relay): per-cell switches for the rejection fence and the lost-reply margin, and report ignored keys

rejectionFence (default on) turns the unhandled-rejection fence off without a roll.
readTimeoutMarginMs (1-60 s) replaces the boot margin per query through pg's query_timeout.
The applied log line and flagsApplied name keys this image does not know.

* feat(relay): runtime-status names the switches and values this image accepts

The flag tool checks a write against it, so it can refuse ticketCheck=enforce (or any key)
on an image that would void or drop it.
2026-10-10 01:54:00 -04:00
Jinwoo Hong e72044a936 fix(relay): a lost database reply ends its connection within statement timeout + 10 s (#26829)
* fix(relay): a lost database reply ends its connection within statement timeout + 2 s

The request pool sets query_timeout to the statement timeout plus 2 s. A
read timeout poisons its transaction: later statements fail at once,
COMMIT never runs (even when the operation caught the error), no ROLLBACK
queues behind the dead statement, and the client is released with the
error so pg-pool destroys it. Autocommit statements destroy their client
too. The server bounds an orphaned idle backend with idle_session_timeout
30 s. A read timeout is answered 503 and never retried, and a renewal
batch that loses its reply fails whole instead of fanning out.

* fix(relay): read timeout margin 10 s, configurable, so routine 5-7 s database stalls never trip it

A whole-VM stall delays replies the server already timed. With a 2 s
margin the read timeout fired inside routine stalls, destroying healthy
clients and failing requests that succeed today. The margin is now 10 s
(ORCA_RELAY_POSTGRES_READ_TIMEOUT_MARGIN_MS), and a real-server test holds
a reply 6.5 s and requires it delivered on the same connection. The fix
level rises to 2 here, with the lost-reply fix.

* docs(relay): name the startup retry that can see a read timeout, and why it is safe

Boot open retries only an unreachable database. The director's startup
cell reconcile does retry a lost reply, which is safe: it re-runs upserts
and limit-change deletes that converge on the same rows.

* fix(relay): answer 503, not 500, when Postgres ends the session mid-request

Staging B1: pg_terminate_backend during /v1/assign returned 500 29 times.
57P01/57P02 and 08000/08003 join the transient SQLSTATEs; pg's
'Connection terminated' and ECONNRESET count only when the database layer
threw them. transaction() still never retries them.

* fix(relay): answer 503 when the pool acquire fails on a dead socket

Staging B1 re-test: a Cloud SQL restart failed PostgresDatabase.transaction's
acquire with 'write EPIPE', answered 500. An acquire failure with EPIPE,
ECONNRESET, ECONNREFUSED, ETIMEDOUT or EHOSTUNREACH is now a pool connect
failure (no statement ran), and acquire failures count as database-layer
errors. EPIPE mid-statement is treated like ECONNRESET.
2026-10-09 18:00:10 -04:00