mirror of
https://github.com/stablyai/orca.git
synced 2026-09-22 08:02:28 +00:00
fix(relay): bound the live-broker wait over a chain of superseding reconciles
The budget bounded the armed-retry chain only. A waiter that kept following fresh reconciles — which the previous two commits made it correctly do — rode that chain with no deadline check at all. MEASURED, on a fake clock: budget 1,000 ms, reconciles landing every 200 ms, opens taking 400 ms and rejecting. Pre-fix the waiter NEVER SETTLED within a 300,000 ms horizon. Post-fix it settles at exactly 1,000 ms, the budget the caller asked for. Reachable in production because `withTransientDemand` reconciles on BOTH acquire and release, so a busy host supersedes a waiter's reconcile faster than opens settle, and the caller rides the chain indefinitely while holding its demand ref. The deadline check is gated on `joinedReconcile` so the contract that matters is preserved: the one open the waiter ARRIVED on is still never cut short, because cutting a slow-but-succeeding open short would fail a pairing that was about to work. The existing "slow first open past the budget still wins" test passes unchanged, which is the assertion that proves the gate. This is the third defect in this loop and all three share one root cause: `await pending` had no escape once the reconcile the waiter captured stopped being the one that mattered.
This commit is contained in:
@@ -229,4 +229,28 @@ describe('relay live-broker wait under interleaving', () => {
|
||||
expect(openBroker).toHaveBeenCalledTimes(1)
|
||||
coordinator.stop()
|
||||
})
|
||||
|
||||
it('keeps the budget over a chain of superseding reconciles, not just over armed retries', async () => {
|
||||
// Every transient-demand acquire and release reconciles, so a busy host can
|
||||
// supersede a waiter's reconcile faster than opens settle. The budget has to
|
||||
// survive that or a caller rides the chain indefinitely holding its demand ref.
|
||||
vi.useFakeTimers()
|
||||
const coordinator = new RelayAuthCoordinator({
|
||||
readContext: async () => context,
|
||||
openBroker: () =>
|
||||
new Promise<CoordinatedRelayBroker>((_resolve, reject) =>
|
||||
setTimeout(() => reject(new Error('slow open')), 400)
|
||||
),
|
||||
onStatus: vi.fn(),
|
||||
random: () => 0.5
|
||||
})
|
||||
coordinator.reconcile()
|
||||
const seen = observe(coordinator.waitForLiveBrokerResult(1_000))
|
||||
const churn = setInterval(() => coordinator.reconcile(), 200)
|
||||
|
||||
await vi.advanceTimersByTimeAsync(2_000)
|
||||
clearInterval(churn)
|
||||
expect(seen()).not.toBe('unsettled')
|
||||
coordinator.stop()
|
||||
})
|
||||
})
|
||||
|
||||
@@ -30,13 +30,21 @@ export async function runLiveBrokerWait(
|
||||
budgetMs: number
|
||||
): Promise<LiveBrokerWaitResult> {
|
||||
const deadline = Date.now() + budgetMs
|
||||
let joinedReconcile = false
|
||||
while (!source.stopped()) {
|
||||
const broker = source.liveBroker()
|
||||
if (broker) {
|
||||
return { broker }
|
||||
}
|
||||
// Why the budget applies only once a reconcile has been joined: the one open
|
||||
// the waiter arrived on is never cut short, but a chain of superseding opens
|
||||
// must not outlive the budget the caller asked for.
|
||||
if (joinedReconcile && Date.now() >= deadline) {
|
||||
return settledResult(source)
|
||||
}
|
||||
const pending = source.reconcile()
|
||||
const superseded = source.authorityChange()
|
||||
joinedReconcile = true
|
||||
// Why unbounded on `pending`: a reconcile always settles (opens carry HTTP
|
||||
// deadlines), and cutting a slow-but-succeeding open short would fail a
|
||||
// pairing that was about to work.
|
||||
|
||||
Reference in New Issue
Block a user