mirror of
https://github.com/stablyai/orca.git
synced 2026-09-22 08:02:28 +00:00
Jinwoo Hong
8e8a9b38ea
* perf(relay): stop indexing the column every control renewal writes relay_assignment_activity_expiry indexes expires_at on relay_assignment_activity_leases, and expires_at is what every control renewal updates: ~471 calls/s, all of them non-HOT because a changed indexed column forbids HOT. The index has one reader, the 30s expiry sweep, which seq-scans the whole 14.8k-row table in under a millisecond. Drop it, and set fillfactor to 70 so a renewal has room for a second row version on its own page. Measured on postgres:16-alpine over 14.8k rows, WAL bytes per renewal and HOT ratio: index, fillfactor 100 (today) 0% HOT 371 B index, fillfactor 70 0% HOT 246 B no index, fillfactor 100 0.5% HOT 298 B no index, fillfactor 70 100% HOT 80 B Both are needed: the index makes HOT illegal, and the default fillfactor leaves no page space to make it possible. Neither statement can use the catalog pre-check as it stood. DROP INDEX IF EXISTS resolves the name before it locks, so once the index is gone it costs a catalog miss and takes no lock on the table - pinned in the lock-target census as the one exempt statement. ALTER TABLE SET does take a lock, so it gets a new 'reloption' target kind that asks pg_class.reloptions for the name=value pair, keeping the invariant that no lock-taking statement reaches a warm boot unchecked. * fix(relay): pre-check the activity-expiry drop and let it defer on a lock timeout The drop had no catalog pre-check, so it was sent on every boot, and a 55P03 from it was fatal: apply-postgres-schema throws on a lock timeout with no retry. On the migration boot that combination is a crash loop. All 28 directors reach the same DROP INDEX at once, it needs ACCESS EXCLUSIVE on a table written ~475/s with lock_timeout at 1s, and a boot that fails restarts the instance to re-queue the same DDL behind the same writers. Two changes: - A new 'index-by-name' lock target. A DROP INDEX names no table, so the existing index check could not serve it; this one resolves by name through the search_path with relkind = 'i', which is how the DROP itself resolves, and skips when absent. DROP INDEX now counts as lock-taking in the census, so it is covered rather than exempt, and IF EXISTS is required the way it is on DROP CONSTRAINT. - A 'schema-deferrable' marker, read from a statement's leading comment. A 55P03 on a marked statement logs orca_relay_postgres_schema_object_deferred and leaves the statement unapplied instead of failing the boot; the next boot re-sends it. Both activity-lease migrations carry it. Everything else keeps the old contract and still fails loudly. SchemaApplySummary gains a deferred count so a boot that skipped work is distinguishable from one with nothing to do. Verified against a real server: with the index present and the table held in ACCESS EXCLUSIVE by another session, both statements defer, the boot completes, nothing is half-applied, and the next boot finishes the job. A warm boot now sends neither statement at all.