mirror of
https://github.com/stablyai/orca.git
synced 2026-09-23 00:02:29 +00:00
Jinwoo Hong
a3046cd27b
* fix(relay): treat pool connect failures as transient, not director faults pg-pool raises connection-acquire failures as a plain Error with no SQLSTATE, so the transient classifier matched only one of the three messages it can produce. The other two reached the routes unclassified and became HTTP 500s, which is what the rollout safety gate counts. The acquire boundary now marks the errors it produces, so "Connection terminated unexpectedly" counts as transient when the socket died during the handshake and stays a hard failure mid-statement, where a retry could repeat a commit whose outcome is unknown. /v1/regions and /v1/admin/evacuation-status gain the transient handling /v1/assign and /v1/resolve already had. * fix(relay): mirror the pool-connect verdict in failure diagnostics The query-failure event's connectionTimeout boolean matched one of the two messages connectionTimeoutMillis can produce, so the 210 dialling timeouts in the last day logged as false and were invisible to the field meant to find them. The pool-connect vocabulary now lives beside the acquire boundary that owns it, and both the router's classifier and the diagnostics read it from there, so the two cannot drift. The event also carries the routing verdict the caller already computed, making "how much of this burst reached users as a 500" one field. * fix(relay): null-safe transient classification and honest transient docs The classifier now runs inside the query catch, where a thrown null or undefined would have turned a database failure into a TypeError that buried it. The diagnostics doc claimed transient maps to a 503 or a 500. Sweeps, startup reconciliation, and admin routes that answer 409 all emit the same event, so counting the false ones over-states user-facing hard failures.
44 lines
2.6 KiB
Markdown
44 lines
2.6 KiB
Markdown
# Relay database failure phases
|
|
|
|
`orca_relay_postgres_query_failed` separates failure to acquire a pooled connection
|
|
(`phase=acquire`) from failure after acquisition (`phase=execute`). It covers
|
|
`PostgresDatabase.query`, including the single-statement control-renewal CTE.
|
|
Statements inside explicit transactions use a different query path and are not
|
|
covered. These events are diagnostic evidence, not a replacement for total SQL
|
|
failure counters.
|
|
|
|
The event contains only an allowlisted error code, a connection-timeout boolean,
|
|
a transient boolean, the operation category (`control-renewal` or `other`), total
|
|
elapsed milliseconds, and pool total/idle/waiting counts at failure. Total elapsed
|
|
time includes acquisition. An acquisition timeout can mean either waiting in the
|
|
queue or establishing a new connection; `connectionTimeout` covers both, and the
|
|
pool counts separate them: a queue wait has waiters, a dial does not.
|
|
Unknown error codes stay `unknown`.
|
|
|
|
`transient` is the classification the request routes act on, not a second
|
|
opinion: true means retryable, false means terminal. It is not a count of HTTP
|
|
responses. Every caller of `PostgresDatabase.query` emits this event, including
|
|
background sweeps, startup reconciliation, and admin routes that map a failure
|
|
to 409, and none of those produces a 503 or a 500. Counting `transient=false`
|
|
therefore over-counts user-facing hard failures; narrow by operation, or join
|
|
against the route's own rejection logs, before reading it that way. A pool that
|
|
cannot hand out a client carries no error code at all, so `code` stays `unknown`
|
|
for that whole class and only these two booleans separate it from a genuine
|
|
fault such as a rejected password.
|
|
|
|
Query text, parameters, error messages, and identifiers are never emitted.
|
|
Successful queries emit no additional event.
|
|
|
|
Use structured GCE logs with `jsonPayload.event="orca_relay_postgres_query_failed"`.
|
|
Compare counts by phase, operation, and code with the same cell's renewal outcomes
|
|
and pool pressure, and with independent PostgreSQL wait samples. Establishing the
|
|
failure phase does not by itself establish why the pool backed up.
|
|
|
|
For production observation, use an immutable image through the same-cap workflow
|
|
on one cell, with fresh monitor evidence and the exact predecessor digest. Verify
|
|
the serving digest and health, then inspect these events during a naturally
|
|
occurring failure. Do not deliberately induce a production database failure.
|
|
Rollback uses the same workflow and predecessor image; no schema or database
|
|
configuration changes are involved. Do not change rehome limits, timeouts, pool
|
|
sizes, or renewal scheduling merely to collect this evidence.
|