Jinwoo Hong
a3046cd27b
* fix(relay): treat pool connect failures as transient, not director faults pg-pool raises connection-acquire failures as a plain Error with no SQLSTATE, so the transient classifier matched only one of the three messages it can produce. The other two reached the routes unclassified and became HTTP 500s, which is what the rollout safety gate counts. The acquire boundary now marks the errors it produces, so "Connection terminated unexpectedly" counts as transient when the socket died during the handshake and stays a hard failure mid-statement, where a retry could repeat a commit whose outcome is unknown. /v1/regions and /v1/admin/evacuation-status gain the transient handling /v1/assign and /v1/resolve already had. * fix(relay): mirror the pool-connect verdict in failure diagnostics The query-failure event's connectionTimeout boolean matched one of the two messages connectionTimeoutMillis can produce, so the 210 dialling timeouts in the last day logged as false and were invisible to the field meant to find them. The pool-connect vocabulary now lives beside the acquire boundary that owns it, and both the router's classifier and the diagnostics read it from there, so the two cannot drift. The event also carries the routing verdict the caller already computed, making "how much of this burst reached users as a 500" one field. * fix(relay): null-safe transient classification and honest transient docs The classifier now runs inside the query catch, where a thrown null or undefined would have turned a database failure into a TypeError that buried it. The diagnostics doc claimed transient maps to a 503 or a 500. Sweeps, startup reconciliation, and admin routes that answer 409 all emit the same event, so counting the false ones over-states user-facing hard failures.
2.6 KiB
Relay database failure phases
orca_relay_postgres_query_failed separates failure to acquire a pooled connection
(phase=acquire) from failure after acquisition (phase=execute). It covers
PostgresDatabase.query, including the single-statement control-renewal CTE.
Statements inside explicit transactions use a different query path and are not
covered. These events are diagnostic evidence, not a replacement for total SQL
failure counters.
The event contains only an allowlisted error code, a connection-timeout boolean,
a transient boolean, the operation category (control-renewal or other), total
elapsed milliseconds, and pool total/idle/waiting counts at failure. Total elapsed
time includes acquisition. An acquisition timeout can mean either waiting in the
queue or establishing a new connection; connectionTimeout covers both, and the
pool counts separate them: a queue wait has waiters, a dial does not.
Unknown error codes stay unknown.
transient is the classification the request routes act on, not a second
opinion: true means retryable, false means terminal. It is not a count of HTTP
responses. Every caller of PostgresDatabase.query emits this event, including
background sweeps, startup reconciliation, and admin routes that map a failure
to 409, and none of those produces a 503 or a 500. Counting transient=false
therefore over-counts user-facing hard failures; narrow by operation, or join
against the route's own rejection logs, before reading it that way. A pool that
cannot hand out a client carries no error code at all, so code stays unknown
for that whole class and only these two booleans separate it from a genuine
fault such as a rejected password.
Query text, parameters, error messages, and identifiers are never emitted. Successful queries emit no additional event.
Use structured GCE logs with jsonPayload.event="orca_relay_postgres_query_failed".
Compare counts by phase, operation, and code with the same cell's renewal outcomes
and pool pressure, and with independent PostgreSQL wait samples. Establishing the
failure phase does not by itself establish why the pool backed up.
For production observation, use an immutable image through the same-cap workflow on one cell, with fresh monitor evidence and the exact predecessor digest. Verify the serving digest and health, then inspect these events during a naturally occurring failure. Do not deliberately induce a production database failure. Rollback uses the same workflow and predecessor image; no schema or database configuration changes are involved. Do not change rehome limits, timeouts, pool sizes, or renewal scheduling merely to collect this evidence.