* fix(relay-ops): roll a cell a wave stranded after its drain
A wave that stops any time after its drain leaves the cell migration-only and
draining on the rollback image, and nothing clears it: the drain flag is a
one-way latch on the running process, and the failsafe restarts nothing. Both
recovery modes then refuse the cell. Apply wants it general and not draining.
Rollback sees the rollback image, reads it as a resume, refuses the draining,
and would not have restarted it anyway.
The image alone cannot separate a rollback that failed after its template apply
from a wave that stopped before one. The restart can: the first left a fresh
process, the second did not. Classify on that, so the cell that never restarted
takes the rolling path instead of the resuming one.
Its template still carries the image it serves, so that is the predecessor its
plan is reviewed against, and a template already moved on to the target is
refused rather than rolled backwards under a stale review. When the reviewed
template is already in place the plan changes nothing, so the MIG is rolled
explicitly on the same replacement policy a template change uses; the existing
incarnation check is what proves the instance came back.
Every other combination of mode, live image, and drain flag keeps the value it
had, held by a census that runs the real block over all nine.
* fix(relay-ops): pin the replacement method on the explicit MIG roll
gcloud persists every rolling-action bound into the group's update policy, and
it defaults the replacement method to substitute on a group with no stateful
config. Passing surge and unavailable without the method would patch the policy
off the declared RECREATE, and the next targeted plan would then carry a MIG
change outside version.0.instance_template, which the plan validator refuses.
Pass all three so the patch is identical to the declared policy, and read the
declared values in the census instead of restating two of them. Dropping the
flag, or moving any of the three in Terraform, now fails the census.
c17's canary stopped at the pre-apply predecessor check with
`runtime predecessor mismatch fields=draining`. The flag is residue: the
previous canary (run 35290908836) drained c17 at 00:26:01, its terraform apply
then failed, and the failsafe re-isolates without restarting the VM, so nothing
cleared it. The same run had passed this very check a second earlier, which is
what proves a parked cell is not draining at rest.
Draining means connections are being shed, and a migration-only cell holds
none, so the flag is not a precondition there. Accept it on entry for that
class only. The replacement VM is still required not to be draining, on every
path, and the incarnation check still proves it was replaced.
Both predecessor checks now read one decision instead of computing the rule
twice, so the assertion and its diagnostic cannot disagree. Every general-cell
and rollback path keeps the value it had; a census test runs the real block
over all eight mode and class combinations to hold that.
c17's canary-apply failed closed at plan validation. Its instance template is
from 2026-08-07 and predates the ORCA_RELAY_CAPACITY_SERVICE_ACCOUNT line that
every cell rolled since already carries, so the plan legitimately added it. The
same-cap validator holds the whole startup script identical before and after
except the image, and that line is not one it excluded, so the wave stopped
with nothing applied.
Pin the line for same-cap-cell exactly as bootstrap-cell already does, and
exclude it from the before/after comparison. The cell may gain it; the pin is
what refuses a roll that drops it or rewrites it to another identity. Both plan
validations in the job now pass the capacity identity the job already requires.
The same-cap contract is otherwise unchanged: any other stale line still fails
closed, and needs a convergence apply before the cell can roll.
A batch-apply wave verified only that the sealed canary named some approved
same-cap cell, so a canary rolled on the migration-only, zero-host, 600-cap
c17 or c18 was accepted as authority for a general 1000/3000-cap batch. The
verify step now hands the batch's own cells to the check, which requires the
sealed cell's entry admission to equal the batch's class.
c17 and c18 hold no hosts and sit outside general admission, so rolling one
displaces nobody. They are the only zero-displacement canary for a new cell
image, but the same-cap wave refused them at the dispatch validator and would
have promoted them to general at the end if it had not.
Add them to the approved list and teach the wave a cell's entry admission
class: the precheck demands the class the cell is declared to serve in, the
restore hands it back that class, the isolate on an already-isolated cell is
asserted to change nothing, and the selector generation advances by 2 for a
general cell and by 0 for a migration-only one. One wave may not mix the two,
because every cell after the first offsets from a single per-wave delta.
Neither cell is a declared regional-rehome source, so its template carries no
rehome trust lines. The source-membership guard now fires exactly when a roll
expects those lines instead of for every US cell, which is the invariant it
was standing in for, and which limits c17 and c18 to rehome protocol 0.
* perf(relay): batch control lease renewals per cell instead of one write transaction per host
Every connected desktop renewed its own control lease with its own single-row
write transaction every 30s. At ~14,000 hosts that is ~470 write transactions
per second fleet-wide, each with its own transaction id, all updating the same
few heap pages of relay_assignments and relay_assignment_activity_leases.
Sampling three onsets at 250ms showed no lock queue and no slow statement:
60-144 backends piled into LWLock:BufferContent and Timeout/SpinDelay inside
that one statement, and Query Insights attributed 152 of 157 seconds of
lightweight-lock wait in the onset minute to it.
The heartbeat now enqueues a due renewal instead of issuing it. A cell flushes
its queue once per second, or as soon as 500 rows are waiting, through one
statement that unnests the parameter arrays and applies the same CTE row-wise.
Concurrent writers drop from the host count to the cell count, and transaction
ids with them. Measured against a 20,000-row table: 1 row 4.1ms, 12 rows
3.3ms, 100 rows 6.6ms, 500 rows 22.7ms.
Per-session semantics are unchanged. Each enqueue still resolves on a renewal
and rejects with the outcome as its message, so the completed-attempt counter,
the staleness guard, and every close path route exactly as before, and one
outcome per row is recorded against the flush latency.
Lock order is (user_id, relay_host_id), the primary key of relay_assignments,
applied in JavaScript and repeated as the statement's ORDER BY. EXPLAIN
confirms LockRows sits above that Sort, so a batch acquires its assignment rows
in one global order. Every writer in the store locks a host's assignment row
before its migration or lease rows and only ever touches one host, so a batch
can only wait on a row a single-host writer holds, never the reverse.
One statement also means one contended row could fail the whole batch, so a
failed batch degrades to the per-host statements it replaced rather than
costing every other host on the cell its renewal.
* perf(relay): batch control lease renewals per cell instead of one write transaction per host
Every connected desktop renewed its own control lease with its own single-row
write transaction every 30s. At ~14,000 hosts that is ~470 write transactions
per second fleet-wide, each with its own transaction id, all updating the same
few heap pages of relay_assignments and relay_assignment_activity_leases.
Sampling three onsets at 250ms showed no lock queue and no slow statement:
60-144 backends piled into LWLock:BufferContent and Timeout/SpinDelay inside
that one statement, and Query Insights attributed 152 of 157 seconds of
lightweight-lock wait in the onset minute to it.
The heartbeat now enqueues a due renewal instead of issuing it. A cell flushes
its queue every second, or as soon as 200 rows are waiting, through one
statement that unnests the parameter arrays and applies the same CTE row-wise.
Concurrent writers drop from the host count to the cell count, and transaction
ids with them. Per-host buffer traffic is unchanged: 30 hits for one row, 25.5
per host at 12 rows, 30.1 per host at 200, against the 28.7 the single-row
statement reports in production.
Per-session semantics are unchanged. Each enqueue still resolves on a renewal
and rejects with the outcome as its message, so the completed-attempt counter,
the staleness guard, and every close path route as before, and one outcome per
row is recorded against the flush latency.
Row locks live until the statement commits, so a batch that waited on a
contended row would hold every other row's lock for that whole wait. The
assignment pass therefore takes its locks with SKIP LOCKED and reports a
contended host as assignment_lock_unavailable, which the registry retries on the
next tick instead of closing the control. That keeps the hold to the statement's
own execution: 11.5ms for 200 rows against a 20,000-row table, and 9.4ms with a
host wedged in a per-host transaction, where a blocking FOR UPDATE spends the
pool's whole 1s lock_timeout and then fails every row in the flush.
An unlocked present_assignment probe separates a host with no assignment row
from one the skip passed over, so a skipped row can never be mistaken for a
missing assignment and close a live desktop.
With no wait on the assignment pass the lock order is only needed for the two
later passes, and it holds: every writer takes a host's assignment row before
that host's lease rows, and a host whose assignment row is held was skipped, so
the batch never reaches its lease. markMigrationTargetRegistered is the one
writer that locks a migration row first, and it takes no further locks.
* fix(relay): answer every row of a control-renewal batch from its own lease update
Review findings on the batched renewal.
Two control leases on one host in one batch made the second report
control_activity_not_found although both were renewed: the assignment UPDATE is
offered the same target row twice, applies one source row and returns one, so
the other row_index never came back. The verdict now reads renewed_lease, which
has a row per input row, and the assignment UPDATE groups per host so it also
stops taking an arbitrary one of the two expiries instead of the later one.
The queue now partitions per (userId, relayHostId) rather than per activity, so
a second control activity for one host opens the next flush instead of sharing
this one. Belt to the statement fix, not a substitute: the store API has to be
right for the rows it accepts.
renewControlActivities recorded no outcome for a one-row flush that threw, and
none at all when every row failed validation, where a mixed batch recorded its
invalid_* rows. Both now record in a finally, the way the single-row path's
finally always did, and the error-to-outcome mapping both paths share is one
function.
The four flush fields the runtime metrics event emits had no log-based metric,
so add them next to the existing controlRenewalLatencyMs* entries. Applying the
Terraform is a separate manual step.
controlRenewalLatencyMsP50/P95/Max now measure a batched row's flush duration
rather than its own statement latency. Left named as they are for history, with
a line at the emit site recording that the meaning changed here.
Added in #21301 on the reasoning that the composite (active, deadline)
index spans all 6.65M rows to find a few hundred live ones. Measured
post-merge against production-shaped history, that reasoning does not
hold: a basis is inserted active and flipped to 0, so the partial index
accumulates one dead entry per deactivation exactly as the composite one
does. Scan buffers are identical to composite-only in every state, 11,099
cold, 2,522 warm, 9 after VACUUM, and the planner picks the composite
index throughout. The extra index costs ~65 bytes of WAL per basis insert,
about 22% more.
The relief is the reaper plus vacuum, which #21301 already ships. Removes
the statement from SCHEMA, restores the plan test to pinning the composite
index by name, and records why a narrower index is not the cure so the
next reader does not re-derive it.
* perf(relay): index and gate the credential cleanup sweeps that seq-scan relay_invites
The credential cleanup ran every 30s in all 23 cells as well as the
director. Both of its relay_invites passes matched columns no index
covered, so each one seq-scanned the whole table inside the maintenance
transaction: 56 calls/min fleet-wide, 129ms and 63ms typical and 57s at
the tail, to return about one row every nine minutes.
Adds partial indexes matching each sweep predicate, gives the cleanup the
same owner as the assignment sweep, and reaps terminal invites after
seven days so the table stops growing for the life of the database.
Every index carries the schema-deferrable marker: an operator builds them
with CREATE INDEX CONCURRENTLY, and the catalog pre-check skips them from
then on.
* perf(relay): index live bases and reap settled connection authorizations
relay_connection_bases is the dominant cost in the cleanup transaction:
195 ms of the 268 ms average, with ~5,800 shared buffer hits per call
even though it already uses relay_connection_bases_active_deadline. That
index spans all 6.65M rows, and only a few hundred are ever active.
Adds a partial index on the live rows alone, and reaps settled rows from
relay_connection_bases and relay_direct_authorizations once their deadline
is more than a day past. Both readers of either table require the row
active/unconsumed and inside its deadline, and every deadline is set at
most 30s past insert, so a settled row can never authorize anything again.
The composite index stays: it is the only one covering active = 0, and it
is what lets the drained reaper learn there is nothing to do from the index
rather than the 1.5 GB heap. Measured at 200k rows, 5 buffers with it and
1,274 without.
* test(relay): accept either bases index in the sweep plan assertion
The negative assertion pinned a planner choice rather than the invariant:
either index keeps the sweep off the 1.5 GB heap, and which one wins on
cost is not something the test should fix. Matches how the same file
already handles the two invite sweep indexes.
Also names the column the authorization reaper actually measures, which
is consumed_at rather than deadline.
* perf(relay): stop indexing the column every control renewal writes
relay_assignment_activity_expiry indexes expires_at on
relay_assignment_activity_leases, and expires_at is what every control
renewal updates: ~471 calls/s, all of them non-HOT because a changed
indexed column forbids HOT. The index has one reader, the 30s expiry
sweep, which seq-scans the whole 14.8k-row table in under a millisecond.
Drop it, and set fillfactor to 70 so a renewal has room for a second
row version on its own page. Measured on postgres:16-alpine over 14.8k
rows, WAL bytes per renewal and HOT ratio:
index, fillfactor 100 (today) 0% HOT 371 B
index, fillfactor 70 0% HOT 246 B
no index, fillfactor 100 0.5% HOT 298 B
no index, fillfactor 70 100% HOT 80 B
Both are needed: the index makes HOT illegal, and the default fillfactor
leaves no page space to make it possible.
Neither statement can use the catalog pre-check as it stood. DROP INDEX
IF EXISTS resolves the name before it locks, so once the index is gone
it costs a catalog miss and takes no lock on the table - pinned in the
lock-target census as the one exempt statement. ALTER TABLE SET does
take a lock, so it gets a new 'reloption' target kind that asks
pg_class.reloptions for the name=value pair, keeping the invariant that
no lock-taking statement reaches a warm boot unchecked.
* fix(relay): pre-check the activity-expiry drop and let it defer on a lock timeout
The drop had no catalog pre-check, so it was sent on every boot, and a
55P03 from it was fatal: apply-postgres-schema throws on a lock timeout
with no retry. On the migration boot that combination is a crash loop.
All 28 directors reach the same DROP INDEX at once, it needs ACCESS
EXCLUSIVE on a table written ~475/s with lock_timeout at 1s, and a boot
that fails restarts the instance to re-queue the same DDL behind the
same writers.
Two changes:
- A new 'index-by-name' lock target. A DROP INDEX names no table, so the
existing index check could not serve it; this one resolves by name
through the search_path with relkind = 'i', which is how the DROP
itself resolves, and skips when absent. DROP INDEX now counts as
lock-taking in the census, so it is covered rather than exempt, and
IF EXISTS is required the way it is on DROP CONSTRAINT.
- A 'schema-deferrable' marker, read from a statement's leading comment.
A 55P03 on a marked statement logs
orca_relay_postgres_schema_object_deferred and leaves the statement
unapplied instead of failing the boot; the next boot re-sends it. Both
activity-lease migrations carry it. Everything else keeps the old
contract and still fails loudly. SchemaApplySummary gains a deferred
count so a boot that skipped work is distinguishable from one with
nothing to do.
Verified against a real server: with the index present and the table
held in ACCESS EXCLUSIVE by another session, both statements defer, the
boot completes, nothing is half-applied, and the next boot finishes the
job. A warm boot now sends neither statement at all.
* feat(relay): pace the drain send during a same-cap cell roll
A same-cap roll drains a cell with graceMs 0, which sends `drain` to all
~800 controls in one pass. Every desktop re-dials on receipt regardless of
graceMs, so the whole cell reconnects inside a second. On 2026-09-16 that
stampede hit a Cloud SQL stall: attaches timed out, each leaving 10 minutes
of late-arrival debt on connection headroom, and placement answered
relay_capacity_exhausted fleet-wide for ~13 minutes.
Spreading the sends spreads the re-dials. `HostSessionRegistry.drain` takes
an optional pacing window and schedules each session's send evenly across
it; admission is fenced for every session up front, and each host keeps its
own full grace after its own send. /v1/admin/drain accepts `paceWindowMs`
(<= 5 min) and echoes what it applied. The same-cap job asks for 120 s, and
the drain-completion wait grew by the same amount. A cell still on an older
image rejects the field, so the deploy script falls back to an unpaced
drain rather than failing the roll.
* fix(relay): scope the drain fence to the hosts already told
Review of the paced drain found two problems, both from treating "this cell
is draining" as one instant when pacing makes it a window.
Timers: the sends queued by a paced drain were neither cleared when a later
drain superseded them nor unref'd. A SIGTERM mid-window left up to 800 no-op
timers holding the event loop open until systemd escalated to SIGKILL. Drain
timers are now tracked, cleared on the next drain, and unref'd, so a retry
re-arms a session's teardown instead of stacking a second one.
Phones: the client fence read the global draining flag, so every phone was
refused for the whole window even though its own host had not been told yet
and was still serving. The director keeps pointing phones at this cell until
their host moves, so they would have looped for up to two minutes. A session
is now fenced when its drain is sent, not when the drain starts, and the
client paths key off that. New control connections and re-attaches stay
fenced globally: nothing new should land on a cell that is going away.
* feat(relay): add a break-glass override for the same-cap monitor gate
Every mutating same-cap wave consumes a fresh 15-minute aggregate monitor
dry-run. When a chronic fault is what the gate freezes on, waiting for a green
window means waiting for the condition the wave removes: the gate froze 44
consecutive times on the recurring Cloud SQL stall the rolling image fixes.
Add `gate-override-reason` and `gate-override-confirmation`
(`SKIP_RELAY_MONITOR_GATE <target-image-digest>`) to the same-cap dispatch. A
valid pair skips only the aggregate evidence download, provenance verification,
and single-use marker. A partial or mismatched override fails closed before any
mutation, in both the caller and the reusable job. Record the actor, reason, and
confirmation in the gate run summary and, for a canary, in the sealed artifact.
The live per-wave preflight still runs. Give it a `--no-monitor-state` source
that takes the expected selector from the dispatch inputs and pins the migration
policy to `strict`, rather than synthesising a state file that would claim a
dry-run it never ran.
Also give `director.instances` the two-consecutive-sample tolerance the cell
probes have: Cloud Run replaces an instance in place, so the count leaves the
[5, 6] band for one sample roughly twice a day, and a deploy overlap raises it
the same way. Min and max share one streak so an alternating count still freezes.
* fix(relay): canonicalise the break-glass preflight membership
The override path parsed the operator's membership with a bare schema parse,
while the live selector read from the director is normalised and the comparison
is an ordered `JSON.stringify`. Unsorted dispatch input would therefore read as
selector drift on a healthy fleet, and the every-configured-cell-exactly-once
check was lost with it.
Normalise through the same `normalizeSelectorMembership` call the monitor CLI
uses when it seals evidence, against the same durable Terraform cell set.
Tests use a collect stub that returns the director's canonical selector rather
than echoing the expected one, so the ordering is actually exercised: unsorted
input must canonicalise, and a duplicated, missing, or unknown cell must be
rejected.
An unreadable sample (collector_failed) now gets the same two consecutive
sample budget per source as an unread signal, so one failed Cloud Monitoring
read no longer restarts the continuous window. monitor_gap keeps zero
tolerance because it means the run itself stopped sampling.
The pre-drain lineage cap moves from 25 to 35 minutes so a 15-minute window
plus one restart still reaches a verdict, and the collector error message is
now logged instead of being swallowed.
The director's /v1/admin/cell-status maps any thrown operation error onto
HTTP 404, so a Cloud SQL pool connect timeout arrived at the ops tooling as
"Relay admin telemetry returned 404" and killed the whole sample. Retry the
admin reads that carry a transient database error, and let the live preflight
spend one of its existing attempts on a thrown collector instead of failing
the wave.
Every admin handler collapsed a thrown error into one status, so a two-second
pool connect timeout answered POST /v1/admin/cell-status with 404. The rollout
tooling never retries a 4xx, by design, so the wave failed on a database that
was briefly out of reach and recovered on its own.
Transient database failures now answer 503 with Retry-After, the shape the
public routes and the region catalog already use. Every other error keeps the
route's existing 404 or 409 mapping.
* fix(cloud): give same-cap waves ten minutes to consume gate evidence
The live preflight rejected monitor evidence older than five minutes, but
the same-cap job only reaches that step about five minutes after the
monitor completes: runner queue, the gate job, and a full-branch checkout.
On 2026-09-17 the first green gate in 44 attempts died at 302 s. The
preflight still takes live samples, so the older baseline is safe.
* docs(cloud): state the ten-minute preflight evidence bound
* fix(relay): treat pool connect failures as transient, not director faults
pg-pool raises connection-acquire failures as a plain Error with no SQLSTATE,
so the transient classifier matched only one of the three messages it can
produce. The other two reached the routes unclassified and became HTTP 500s,
which is what the rollout safety gate counts.
The acquire boundary now marks the errors it produces, so "Connection
terminated unexpectedly" counts as transient when the socket died during the
handshake and stays a hard failure mid-statement, where a retry could repeat a
commit whose outcome is unknown.
/v1/regions and /v1/admin/evacuation-status gain the transient handling
/v1/assign and /v1/resolve already had.
* fix(relay): mirror the pool-connect verdict in failure diagnostics
The query-failure event's connectionTimeout boolean matched one of the two
messages connectionTimeoutMillis can produce, so the 210 dialling timeouts in
the last day logged as false and were invisible to the field meant to find them.
The pool-connect vocabulary now lives beside the acquire boundary that owns it,
and both the router's classifier and the diagnostics read it from there, so the
two cannot drift. The event also carries the routing verdict the caller already
computed, making "how much of this burst reached users as a 500" one field.
* fix(relay): null-safe transient classification and honest transient docs
The classifier now runs inside the query catch, where a thrown null or
undefined would have turned a database failure into a TypeError that buried it.
The diagnostics doc claimed transient maps to a 503 or a 500. Sweeps, startup
reconciliation, and admin routes that answer 409 all emit the same event, so
counting the false ones over-states user-facing hard failures.
* fix(relay-ops): let the pre-roll gate ride out chronic production noise
The 15-minute pre-drain dry-run froze 39 times out of 39 on conditions
that have nothing to do with the roll it gates:
- A cell probe is one HTTP round trip from one runner. When the Asia
cells' readiness SQL probe times out behind a saturated pool, the load
balancer answers "no healthy upstream" for ~30 s and the gate froze on
a single sample. Cell probe signals now need more than
cellProbeToleranceSamples consecutive failing samples to freeze;
absorbed blips are recorded in the state artifact. Director and auth
probes keep zero tolerance.
- directorErrors 3 -> 15. Measured non-503 5xx per rolling five minutes
over the 24 h to 2026-09-17: p90 3 / p95 5 / p99 9 / max 52. The old
bar sat on the p90 and froze 29% of gates.
- cloudSqlBackends 250 -> 320. Measured latest-sum over the same 24 h:
p95 212 / p99 262 / max 282. The old bar sat under the observed peak
and froze 22% of gates.
Failure codes are unchanged so downstream matchers keep working, and the
trusted evidence scripts are untouched.
* fix(relay-ops): key probe tolerance by cell and extend it to live preflight
Three review findings on the cell-probe tolerance:
- The streak was keyed per signal, so a cell alternating between slow
(latency over bar) and down (health/ready 0) held every individual
streak at one and never reached the tolerance. A continuously unhealthy
cell passed the gate. The streak is now keyed by cell id, so one cell's
health, ready and latency readings share it.
- The live preflight runs one sample before every mutating wave and
retried only on freshness codes, so the same Asia blip could still fail
a wave there. It now re-samples per-cell probe breaches on the same
tolerance, spaced the existing interval. Director and auth probes still
fail the wave on the first bad sample, as does any non-probe threshold.
- docs/relay-incident-monitor.md still stated the old bars. Updated the
threshold table, the 400-connection ceiling text, and the superseded
2026-08-26 and 2026-09-12 entries, and added a dated 2026-09-17
recalibration entry.
Also pins the resumed-state case: a state file carrying a full streak now
has a test proving it freezes on the next bad sample.
Trusted evidence scripts remain untouched.
* fix(cloud): pin the asia cell database pool in the same-cap plan validator
Raising `database_pool_max` from 10 to 16 for production-gce-c27, c28 and c29
made every same-cap roll of those three cells fail closed at plan validation.
The cell startup template emits `ORCA_RELAY_DATABASE_POOL_MAX` only for a cell
whose region differs from the root region or whose pool is off the default, so
the asia cells carry that line while the us-central1 cells do not. The plan
validator requires the before and after startup scripts to normalize to the
same text, masking only the lines it independently pins to a reviewed value.
The pool line was neither masked nor pinned, so the live template's `'10'` and
the plan's `'16'` were read as unreviewed drift.
The validator gains an optional `--database-pool-max`, accepted in
`same-cap-cell` mode alone. When it is supplied the after-script must contain
exactly that pool line and the line is masked from the equality check; when it
is not supplied the after-script must contain no pool line at all. Masking
without the pin would have removed the guard rather than moved it.
The same-cap job resolves the expected pool next to the hard cap, cross-checks
it against the committed `relay_gce_cells` map (asserting the default 10 for
the us-central1 cells), and passes the flag to both validator invocations only
for the cells that emit the line.
* test(cloud): require the pool pin for a line the live template already carries
* infra(relay): raise asia-east2 cell pools to 16 and retire four idle cells
The three asia-east2 cells sit 176 ms from the Cloud SQL instance in
us-central1. Server-side statement time there is 0.2 ms, so a pool slot is
held by the round trip, not by the query. At a pool of 10 they measured
94-156 waiters and 2 s waits, and client accepts ran a ~4 s p95 against
222-646 ms in us-central1. Raising those three pools to 16 is the agreed
first step; every other cell stays at 10.
c4 and c5 join the committed fence set. Both are existing-only capacity the
admission selector can never place on again, they carried ~1 connection each
on 40-day-old images, and each still holds 10 Postgres connections. The fence
set is the prerequisite the fence-source workflow confirms before it drains
and attests a cell; it is not itself the resize.
c17 and c18 are not fenced here. They are migration-only, and the runbook
requires retire-migration-cell to move a migration-only cell to existing-only
through a generation-bound selector CAS before it can be fenced. Terraform
cannot express that step.
The Cloud SQL consumer contract carried two stale numbers: auth at 2 instances
when production has run a cap of 20 since 2026-09-04, and a 400-connection
ceiling when the live instance reports 500. Both are corrected, and the budget
now asserts its headroom in two named gates instead of one aggregate boolean.
Those gates fail: auth alone accounts for 200 configured connections and a
215-connection rollout overlap, so the operating maximum is 713 against a
usable ceiling of 490. Nothing here caused that, and no pool was lowered to
hide it.
* infra(relay): move the Cloud SQL contract correction out of this branch
The contract correction (auth at its real 20-instance cap, the measured
500-connection ceiling) makes the budget gate fail for reasons that have
nothing to do with asia pools or fenced cells, and it held this branch red.
It moves to its own branch where the failure is the subject.
production-cloud-sql-app-consumers.json returns to main unchanged. The budget
test keeps main's single gate and only repins the cell figure that this branch
genuinely moves: 230 -> 228, being +18 for three asia pools at 16 and -20 for
fencing c4 and c5. Against main's 400-connection model that leaves an operating
maximum of 383 under a usable ceiling of 390.
* infra(relay): move the c4/c5 fence entries out of this branch
Terraform now sets a cell's MIG target size directly from relay_gce_fenced_cells
(relay-gce-cells.tf); the lifecycle ignore that used to protect operational
target_size drift is gone. So a fence entry sitting on main ahead of its
fence-source run is a standing instruction that any apply reaching that cell may
execute without the documented drain and attestation. Keeping the entry in the
same merge as an unrelated pool change widens that blast radius for no reason.
The two entries move to their own branch, to be merged immediately before
fence-source runs for c4 and then c5. This branch keeps the multi-line reflow of
the list, which makes that later diff two added lines instead of a rewritten one.
The cell figure in the budget test follows: 230 + 18 for the three asia-east2
pools at 16, with no fenced-cell subtraction. That is 403 operating against a
usable ceiling of 390, so the headroom gate now fails by 13. It fails against a
ceiling of 400 that is itself wrong; the instance reports 500. See the PR body.
* infra(cloud-sql): record the measured 500-connection ceiling
The budget's usable ceiling came from maxConnections: 400, described as the
tier default. It is a tier default, since no max_connections flag is set, but
the instance does not report 400. SHOW max_connections on it returns 500,
measured 2026-09-16.
On main the model sat at 385 against a usable ceiling of 390, five connections
of margin, so raising the three asia-east2 pools by 18 failed the gate by 13
against a ceiling that was never checked. Against the measured one it is 403
against 490, clearing by 87.
Only the ceiling and its source note change here. auth stays recorded at 2
instances, which is also wrong; PR #21165 corrects it, and with the true auth
figure the budget is over by 225 for reasons that have nothing to do with these
pools.
* test(cloud): state the cell pool arithmetic literally in the budget pin comment
* fix(relay): serve readiness from last-known-good during auth or SQL blips
The load balancer health check hits /ready, which re-probed the auth JWKS
endpoint and Postgres on every poll and reported not-ready on the first
failure. On 2026-09-16 an auth outage therefore took every cell out of the
load balancer within ~30s and dropped every connected host, even though the
token verifier caches keys in process and kept verifying tokens.
/ready now remembers when each dependency last answered and keeps reporting
ready while the failed one stays inside a grace window
(ORCA_RELAY_READINESS_GRACE_MS, default 15 minutes, 0 disables). A process
that has never succeeded still gates on the real dependencies, so cold boot
is unchanged. Grace answers carry degraded plus the failure reason on the
existing readiness observation, and entering or leaving grace logs once.
MIG autohealing still uses the dependency-free /health endpoint.
* fix(relay): split readiness grace per dependency and probe both every poll
Review follow-ups on the last-known-good readiness window.
An unset environment variable arrives as an empty string, which z.coerce
reads as 0, so the single ORCA_RELAY_READINESS_GRACE_MS would have switched
the window off instead of falling back to its default. The two replacement
variables preprocess '' to undefined.
JWKS and SQL now get separate windows and separate clocks:
ORCA_RELAY_READINESS_JWKS_GRACE_MS defaults to 15 minutes, and
ORCA_RELAY_READINESS_SQL_GRACE_MS to 3 minutes. Each cell is its own load
balancer backend, so failing readiness never re-routes a host, it only makes
that hostname unreachable, and a host that lands on a SQL-dead cell gets
WRONG_CELL and is re-placed by the director. Three minutes rides a Cloud SQL
failover without hiding a per-cell fault for a quarter of an hour.
Both dependencies are probed on every poll. A JWKS failure used to
short-circuit the SQL probe, which let the SQL clock age with no evidence
behind it. Grace transitions are emitted per dependency, so JWKS recovering
while SQL fails logs both sides instead of nothing.
/ready keeps its 200 and its {ok:true} body when healthy, and adds
degraded plus the dependency list when the answer comes from a window.
* fix(relay): skip boot-time DDL when the catalog already has the object
CREATE INDEX IF NOT EXISTS and ALTER TABLE ADD COLUMN IF NOT EXISTS take
their relation lock before the server evaluates the existence test, so a
boot on an already-migrated database still joins the lock queue. Relation
locks are granted in queue order, so every writer queues behind it.
The shared runner now asks pg_catalog whether the index or column is
already there and skips the statement when a row comes back, and 55P03
is no longer retried by default: with the pre-check ahead of it, a lock
timeout means the object is genuinely missing and each retry re-enters
the queue. Push keeps the old retry behind an explicit option.
* fix(relay): tie the index pre-check to its table and fail on an unreadable target
Three defects found in review of the auth reference implementation:
- The catalog query matched an index by name inside the table's namespace
without checking it belonged to that table. Index names are unique per
schema, not per table, so a same-named index on a sibling table answered
yes and the real index was skipped forever. Added i.indrelid = t.oid.
- Lock-target derivation read a keyword sitting in an identifier position as
the object name: CREATE UNIQUE INDEX CONCURRENTLY ON t(c) yielded the name
CONCURRENTLY, and ADD COLUMN IF NOT EXISTS with no column yielded IF. A
wrong target is worse than none, so keywords are now excluded and an index
or column statement whose target cannot be read throws at boot with the
statement text instead of falling through to the lock path.
- A concurrent-create collision retried the CREATE INDEX, taking SHARE on the
table again for an object another director had just finished creating. The
catalog is re-asked instead and a present object counts as skipped.
* fix(relay): pre-check constraint swaps so a warm boot sends no DDL at all
The two ALTER TABLE constraint statements were the last lock-taking
statements without a pre-check, so every boot still took ACCESS EXCLUSIVE
on relay_region_rehome_attempts twice.
A lock target now carries the catalog answer that means there is nothing
left to do. ADD CONSTRAINT skips when pg_constraint already names it; DROP
CONSTRAINT IF EXISTS is the inverse and skips when it does not, because
nothing to drop is nothing to do. The match is by name only: the CHECK body
is generated from RELAY_REGIONS, so comparing it would re-run the swap on
every region change. Changing a definition under the same name is an
operator migration, and the rule comment beside SCHEMA says so.
A bare DROP CONSTRAINT gets no target and throws at boot, because skipping
it would swallow the undefined_object the server is supposed to raise.
The census invariant is now that every lock-taking statement has a
pre-check, with no exceptions, and the warm-boot Postgres test asserts zero
statements sent rather than two.
* fix(relay): refuse a multi-action ALTER TABLE instead of pre-checking its first action
`ALTER TABLE t ADD COLUMN IF NOT EXISTS a TEXT, ADD COLUMN IF NOT EXISTS b
TEXT` derived the target for `a` alone, so once `a` existed the whole
statement was skipped and `b` was never added. The first subcommand parses,
so neither the parse throw nor the census caught it.
A lock-taking ALTER TABLE with a comma outside parentheses, quotes and
comments now throws at boot. One action per statement, or no pre-check is
possible. Commas inside a parenthesised type, a CHECK body, a quoted
default or a comment are unaffected, and push's 18 statements still parse.
* fix(relay): strip every comment before classifying, fold catalog names, count brackets
Four findings from the bot reviews on #21147:
- A comment between two keywords (ALTER TABLE t ADD /* note */ COLUMN c
TEXT) was invisible to both the classification regexes and the must-parse
shapes, so the statement got no target AND no throw and ran with no
pre-check. Every comment is now stripped quote-aware before classification,
nested block comments included. The server is still sent the original text.
- hasTopLevelComma counted parentheses but not square brackets, so
ADD COLUMN c bigint[] DEFAULT ARRAY[1, 2] read as two subcommands and
failed the boot.
- bareIdentifier split a qualified name on '.' regardless of quoting, so
"a.b" became b", and it kept the written case while Postgres folds an
unquoted identifier to lower case before storing it in relname, attname
and conname. The name is now tokenised quote-aware and folded, with the
qualified table text still passed to to_regclass as written.
- sqlWithoutLeadingComments is renamed sqlWithoutComments to match.
Relay's 74 statements and push's 18 all still parse, and no relay target
name changed: every identifier there was already lower case.
* fix(relay): treat a dollar-quoted body as opaque in both scanners
A comment marker, comma, parenthesis or bracket inside `$$...$$` or
`$tag$...$tag$` is text. The closing delimiter has to match the opening tag
exactly, so an inner `$$` inside a `$tag$` body is more text rather than the
end, and a tag cannot start with a digit, which keeps a `$1` placeholder
from reading as an opener.
Relay's pg_stat_statements DO block is the only dollar-quoted statement in
the schema, and it now survives the stripper byte-identical. A test asserts
that against the real statement.
The fleet safety gate returned database_pool_pressure whenever the
Math.max of database_pool_waiters_max or database_pool_wait_ms_max
across every general cell crossed 16 waiters or 250ms. Measured
2026-09-16, the asia-east2 cells breach continuously at 94-156 waiters
and ~2000ms while their server-side execution is 0.2ms, which is a
client pool too narrow for a 176ms round trip rather than database
distress, and the us-central1 cells breach in bursts on about a third of
polls. Worse, the bar flaps: the pre-check passes, the commit re-check
reads fresh rows seconds later and trips, and that path durably disables
the control instead of merely deferring.
Drop the pool check from the fleet gate. Pool pressure stays a per-cell
exclusion in regionalRehomeCellSafetyIsClean, which already drops a
breaching cell as both source and target on selection and again on the
commit path. The fleet bars that remain (stale monitoring, sql failure
storms, control-recovery failures, reconnect storms) all signal
database-wide distress. Nothing cells publish, no stored row and no
exported constant changes.
* feat(relay): count failed cell-inventory lock acquisitions
The cell inventory lock is taken NOWAIT, so contention errors with 55P03 and
retries instead of waiting. CellInventoryHoldSamples.record only runs after a
successful acquisition, so the hold metrics were structurally blind to the
dominant failure mode: production showed ~65 failed fleet-wide acquisitions per
minute while cellInventoryHoldMsMax read a benign 53ms mean.
Count failures next to the holds and publish them as cellInventoryLockUnavailable
in orca_relay_runtime_metrics. Drained on both the commit and the rollback path,
since a 55P03 rolls its transaction back.
* fix(relay): separate request-path lock timeouts from sweep deferrals
Review caught that the first counter only incremented under failIfUnavailable,
which is the sweep mode. Background sweeps take the inventory NOWAIT and
re-derive a skipped candidate next tick, so those deferrals are by design and
already reported as orca_relay_sweep_cell_inventory_busy. The request path uses
a bounded lock_timeout instead, whose expiry raises the same 55P03 without
NOWAIT and was not counted at all -- so the metric measured only the benign
population and missed the user-visible one.
Split them: cellInventoryLockUnavailable for NOWAIT deferrals,
cellInventoryLockTimeouts for expired bounded waits. Production over 30 minutes
shows why the distinction matters -- roughly 1,200 fleet-wide sweep deferrals
against roughly 10/min request-path timeouts.
Adds transaction-path coverage for both drains, which were previously unpinned.
Timeouts count per attempt, not per request, since 55P03 is retryable.
* fix(relay): publish the cell-inventory lock metrics to Cloud Monitoring
google_logging_metric.relay_snapshot only creates metrics for fields listed in
relay_runtime_metrics, and the cellInventoryHold* fields were never added when
the hold telemetry landed. They have been log-only since, so nothing could
alert on the lock and the contention stayed invisible in exactly the way the
telemetry was meant to prevent.
Maps the three hold fields and both new failure counters.
Also corrects the field comment: the split is by wait policy, not by caller.
assignOnce takes the inventory fail-fast on its first placement attempt, so
request-reachable sites land in cellInventoryLockUnavailable too; that lane
reads as contention pressure, and the expired bounded wait is the stall lane.
The production change (single insertion-order scan of the session inventory,
reused by the unfenced leg) landed in #20219. This carries the regression
coverage for it: a 1,000-session differential suite that counts iterator visits
and pendingConns.has probes against the pre-change two-find oracle, ordering
under duplicate connection IDs, and attach-ownership tests on the client-accept
path. Folds host-session-owner-scan.test.ts into that suite.
* fix(relay): keep failed rehome polls out of the durable failure budget
The regional rehome worker polls claimRegionalRehome about once a second.
Any error thrown before an attempt was claimed - in practice a director pool
timeout on the pre-claim control read, 52-74 a day against a pool of 3 - was
charged to relay_region_rehome_worker_state.consecutive_failures, which
durably disables the control at three. That counter only ever resets on a
drain receipt, so while the control is disabled it never resets: production
sits at 1068 and still climbing. Enabling the control leaves the stale
counter in place, so the next pool timeout latches it straight back off.
That is what ended the 2026-08-28 enable after ten minutes.
- A poll that never claimed an attempt drained nothing, so it no longer feeds
the dispatch-failure budget and logs .._poll_failed instead of
.._dispatch_failed. recordRegionalRehomeWorkerFailure had no other caller
and is removed.
- Enabling the control clears consecutive_failures and paused_until, so a
budget spent under a previous enable cannot kill a fresh one. The dispatch
interval in next_dispatch_at is deliberately left alone.
- The budget's auto-disable now emits
orca_relay_regional_rehome_failure_budget_disabled, matching the existing
.._safety_disabled precedent. It wrote no event before, which is why this
went unnoticed for two weeks.
No change to region selection, the candidate query, or host eligibility.
* fix(relay): serialize rehome failure accounting with control updates
The unit tests cover parseRelayHostCapabilities, the sendHelloAck gating, and the
header literal separately, but nothing joined them: a typo in the header name
read off the upgrade request passed the entire suite. This drives a real control
upgrade carrying the header, leaves an invite connection pending, and asserts the
rebound control's ack. Renaming the header the server reads fails it.
* fix(relay): bound control RTT samples per ping and per flush window
An authenticated host chose how many round-trip samples a cell recorded: every
pong carrying a recent plausible `t` was forwarded to the process-wide window,
which grew unbounded until the 30s flush copied and sorted it for percentiles.
Time a pong only when it echoes the `t` of the ping still outstanding on that
session, so a flood yields at most one sample per ping the cell actually sent.
A pong that lost the race to the next ping is dropped for timing but still
counts as proof of life for the silence watchdog. Bound the process-wide window
with a 1024-sample reservoir (Algorithm R) so the percentiles stay unbiased,
keep `controlRttSamplesDelta` meaning round trips observed, and publish
`controlRttSamplesDroppedDelta` for the ones the reservoir did not keep.
Replace the leak guard's blanket `"credential":` string rewrite with an exact,
path-scoped rename of the two schema keys that spell a policed word, and make
the guard case-insensitive now that nothing legitimate trips it.
Follow-up to #19232.
* test(relay): prove the RTT reservoir samples the whole window
The cell announces a connection with a single conn-open. When the desktop's
control socket dies mid-accept the phone waited out the 10s attach deadline and
was closed HOST_OFFLINE, even though the desktop was online. host-hello-ack
already restates those connections in pendingConns, but only by connId and
connTicket, which is not enough for the desktop to dial: kind and relayDeviceId
decide the pairing authority a connection carries and the E2EE device binding,
so neither may be guessed.
The cell now states kind and relayDeviceId on each pending entry, but only to a
host that advertised it can read them: a shipped host parses those entries
strictly, so an unannounced key fails the whole ack parse and kills a working
control. The advertisement rides the control upgrade as
x-orca-host-capabilities, not host-hello, because HostHelloSchema is strict on
the cell too and any new hello key is refused by every already-deployed cell.
The capability is keyed by socket, not by session: a rebind can land a successor
whose decoder is older or newer than the one that opened the session, and the
ack must follow the socket that will actually read it.
With no capable host in the fleet the emitted ack is byte-identical to today's.
The desktop half that consumes the new fields is #19238.
`relay_region_rehome_source_cell_ids` listed only the 16 US cells, and that
list is the sole thing that stamps ORCA_RELAY_REHOME_DIRECTOR_SERVICE_ACCOUNT
and ORCA_RELAY_REHOME_AUDIENCE into a cell's startup script. A cell reports
regionalRehomeProtocol 1 only when both are present, so c27-c29 have always
reported 0. That leaves them ineligible as rehome sources and, once the worker
is bidirectional, as targets too, which strands the US desktops homed there.
This is a prerequisite only. Merge and roll it ONLY AFTER the bidirectional
rehome director change is deployed. Two live gates still hard-code the primary
region and would reject an Asia source no matter what the template stamps:
`cloud/apps/relay/src/app.ts` line 610 fails the trust probe with 409 when the
source cell's region is not RELAY_DEFAULT_REGION, and
`cloud/apps/relay/src/assignment-store.ts` line 5476 skips such a cell as
source_ineligible during rehome source selection. The bidirectional lane
removes both.
The topology check asserted every source sits in the primary region. That
mirrored those two gates rather than protecting anything Terraform owns, so it
is now advisory: it requires only a configured, unfenced cell with an explicit
connection limit, and the comment records that region eligibility belongs to
the director's own source and target predicates. Every cell's region is
already constrained by the assert above it.
The same-cap census test cross-checked membership against us-central1. Every
reviewed serving cell now carries the trust, so it asserts protocol 1 for all,
plus one non-source cell to keep the validator's protocol-0 branch covered.
Roll sequencing, because this apply is not self-contained:
- After the apply the Asia templates carry the two rehome lines, and the
`unexpectedRehome` rule at `cloud/dev/scripts/validate-relay-capacity-plan.mjs`
lines 243-247 rejects a protocol-0 plan that contains them. So c27-c29 have
no dispatchable protocol-0 same-cap roll until the director gate is gone or
this is reverted.
- The same-cap job runs the per-host trust probe after isolate, drain, and the
targeted apply. A 409 there leaves the cell serving but isolated and
migration-only, which is what happened to c13 on 2026-09-06.
- The only safe path: deploy the bidirectional rehome director, then dispatch
`Deploy Relay Production Same-Cap` canary-apply for one Asia cell with
target-rehome-protocol 1 and rollback-rehome-protocol 0, then batch-apply the
remaining two. That job runs its own targeted template and MIG apply.
- Never reach these cells with an untargeted root apply. The current plan
carries 60 changes and 50 destroys of unrelated standing drift.
* feat(relay): alert on far-cell placement and skewed region hints
US desktops were homed on asia-east2 cells for weeks in 2026-08 with every
existing relay alert green. Roughly 226 of 332 hosts on those cells were
non-APAC, and a phone connect took ~10 s there against ~0.6 s in region, but
nothing in Cloud Monitoring could see distance: the connection, queue, heap,
and SQL bars all measure a cell's own health, which was fine.
Three policies close that gap. Two read distance per cell, from the accept
and control-RTT timing added in the parent commit: phone-accept p95 above
2 s, and control ping p50 above 150 ms. The third reads the cause fleet-wide,
as the asia-east2 share of the region hints desktops send the director, so a
mis-picking client probe is visible before it lands anyone on a far cell.
All three are MQL rather than the metric filters the other relay policies
use. Every runtime metric is a DELTA DISTRIBUTION, and a filter condition can
only align one with a percentile; each alert needs the sum of the extracted
values as a volume floor so a sparse window cannot page. None of these
metrics exists in the project yet, so what was checked against production is
the query shape: the same MQL run over existing metrics of the same kind.
The skew denominator needs one log-based metric per hint key, so
`requestedRegionsDelta` now has one per relay region plus the unhinted
bucket. Those ride the existing snapshot metric family, which adds map
entries without touching the live metrics. A ratchet test pins the key list
to relay-contract's RELAY_REGIONS: a region added there without a metric
would shrink the denominator, so the test fails rather than letting the share
quietly inflate.
* fix(relay): compare hinted regions against placed ones, not a fixed share
Review found the skew alert inverted at both ends. A fixed 40% bar on the
asia-east2 share of region hints was silent through the exact broken state it
was written for, and would page forever once the desktop probe is fixed and
the genuine APAC share rises past it. An absolute share cannot separate those
because it has no reference point.
The hint share now has one: the share of assignments the director actually
placed in that region during the same hour. Measured over twelve hours on
2026-09-07, while the probe was still mis-picking, asia-east2 was 33.8% of
33,800 hinted requests and 7.9% of 45,364 assignments. That is a 4.27x
divergence and a 25.9-point gap, so the alert fires above 2x and 15 points,
inside the broken state and outside a healthy one. Both bars must hold: the
ratio alone blows up on tiny placement counts, the gap alone misses a
proportionally large skew at low volume. The reviewer proposed either bar
alone; requiring both keeps each one meaningful and still clears today's
numbers with room.
`unhinted` requests leave the denominator. They were 27% of all requests, so
a client that always sends a hint would move the number from 21.9% to 35.0%
with no behaviour change at all.
The comparison needs per-region placement counters, so `selectedRegionsDelta`
gets log-based metrics alongside the requested ones. Rather than extract four
hyphenated map keys through quoted field paths, which nothing in the project
does and which cannot be checked without applying, the relay now also
publishes flat `requestedRegion<Region>Delta` and `selectedRegion<Region>Delta`
fields next to the untouched maps. They are emitted as zeros in every
interval, so no series can drop out of the alert's inner join in an hour with
no asia placements, which is exactly the hour the skew is worst. Additive
only: metricVersion is unchanged, the maps still carry anything outside the
catalog, and the emitter's leak guard still passes.
Two corrections to what the previous commit claimed. None of these metrics
exist in the project yet, so the code, the doc and this message now say what
was actually checked against production: the query shapes, run over existing
metrics of the same kind. And the control-RTT policy records that EU desktops
on us-central1 sit at 100-130 ms, so a European-heavy cell can approach the
150 ms bar while correctly homed.
The skew alert will stay lit after a client fix until the backlog is rehomed.
Sticky assignment never re-consults the hint, so a desktop already on an asia
cell keeps landing there whatever it now asks for. The policy description and
the doc both say so, so nobody reads a slow clear as a failed fix.
* fix(relay): cross-multiply the skew bars so a zero placement share still fires
`hint_share / placement_share` is undefined in the hour that matters most.
When the director placed nobody in the region, MQL returns no rows for either
0/0 or x/0, so the series disappears before the gap and volume clauses run and
the alert stays silent. That hour is not hypothetical: it is every desktop
asking for a region while the director puts nobody there, which is what a
drained, fenced, or full region looks like, and it is the most extreme skew
the alert can see.
The condition is now cross-multiplied, `hint_share > 2 * placement_share`,
which is well defined at zero. Both forms were run read-only against
production surrogates chosen so the placement denominator is exactly zero:
the ratio form returned no rows, the cross-multiplied form returned the series
with the condition true on every point. A second surrogate pass with a tiny
hint share returned the series with the condition false, so the gap clause
still suppresses the healthy shape rather than the query silently matching
everything.
The flat field names are no longer derived on either side. Terraform title
cased each dash-separated part and the emitter upper cased each part's first
character, so the ratchet had to pin two source expressions by regex, which a
reformat would break and which never compared the actual rendered names. Both
sides now declare a literal map, relay-contract's
RELAY_REGION_METRIC_SEGMENTS and Terraform's relay_region_field_segments, and
the test compares the two declarations against each other and against the
expected names. `satisfies Record<RelayRegion, string>` makes a region added
without a segment a compile error rather than a silent gap in the alert's
denominators.
Both ratchets were checked by mutation: a wrong Terraform segment, a contract
region with no Terraform entry, and a revert to the ratio form each fail the
node test, and the new region fails the contract build.
Jinwoo Hong
Neil
Neil
OrcaWinandm4air