test(reliability): benchmark terminal flow control

This commit is contained in:
OrcaWin
2026-07-22 22:15:59 -07:00
parent de272bf65c
commit 871f416c4c
+50 -6
View File
@@ -4608,16 +4608,18 @@
"macos"
],
"coveredProviders": [],
"coverageNotes": "Local macOS evidence over the runtime-RPC stream budgets on main@1282f5c2d. PR #5824 adds a platform-neutral mobile decision gate proving chat-covered terminal streams pause and resume only after the mounted WebView is ready; live Android restore evidence remains required. The pending stack adds byte-exact 512KB/2MB/256KB/48KB budget assertions; legacy JSON subscribe parity remains undecided.",
"coverageNotes": "Local macOS evidence covers runtime-RPC stream budgets plus paired-renderer parse/discard credit. Deferred credit is shared by local and remote transports, batches ACKs at 192 KiB or 4 ms, grows per-stream windows from 512 KiB to 2 MiB and aggregate windows from 2 MiB to 8 MiB, bounds queued output to 256 KiB per stream, and caps each multiplex connection at 32 active or pending streams for an 8 MiB aggregate pending-output ceiling. Deterministic tests cover replay ordering, stale generations, malformed frames, hidden panes, queue eviction, disposal, send/recovery failure, repeated pending-slot replacement, reconnect, and round-robin fairness. The opt-in benchmark covers 1/20/100 ms RTT and 1/4/8 viewers, exact protocol-frame allocations, scheduler CPU, and measured @xterm/headless parser CPU/retained heap. Live Android restore evidence, browser/WebGL parser measurements, and legacy JSON subscribe parity remain required.",
"motivatingLinks": [
"https://github.com/stablyai/orca/pull/6951",
"https://github.com/stablyai/orca/pull/6955",
"https://github.com/stablyai/orca/pull/7009"
],
"invariant": "Runtime and mobile terminal subscriptions must cap initial snapshots, live output buffered while snapshots load, chunk sizes, and batch sizes; a terminal covered by native chat must have no live output subscription and must restore from fresh scrollback when revealed, while preserving output order, input locks, resize/driver events, and fallback parity or explicit fallback deprecation.",
"oracle": "The current executable slice asserts mobile initial snapshots downgrade until they fit <=512KB, requested binary snapshots downgrade until they fit <=2MB, binary live output queued while the initial snapshot loads stays <=256KB while preserving the newest tail, large binary output is split into <=48KB frames, output bursts are coalesced before emit, aborts do not register stale listeners, and stale mobile resize re-stream completions are dropped. The mobile native-chat decision test asserts an active stream pauses while covered and resumes only for a ready active terminal. JSON fallback parity and live Android scrollback restoration remain explicit gaps.",
"invariant": "Runtime and mobile terminal subscriptions must cap initial snapshots, live output buffered while snapshots load, chunk sizes, batches, and aggregate in-flight credit. ACK means the renderer parsed the bytes or intentionally discarded them; receipt-time ACK is forbidden. Every replay, stale-generation, malformed-frame, hidden-pane, eviction, disposal, error, and reconnect path must settle credit exactly once so streams neither leak memory nor stall. A terminal covered by native chat must restore from fresh scrollback when revealed, while preserving output order, input locks, resize/driver events, fairness, and fallback parity or explicit fallback deprecation.",
"oracle": "Assert mobile initial snapshots downgrade until they fit <=512KB, requested binary snapshots downgrade until they fit <=2MB, live output queued while snapshots load stays <=256KB per stream, large output splits into <=48KB frames, and output bursts coalesce. Feed paired output through the xterm parse callback and prove ACK is deferred until parse or intentional discard, then inject stale generation, malformed/transformed frames, replay failure, queue eviction, hidden panes, pane disposal, ACK send failure, recovery serialization failure, and reconnect races; assert ordered replay and exactly-once credit settlement. Fill the aggregate window across bulk and interactive streams, ACK once, and prove round-robin progress. Run the opt-in 64 MiB/viewer RTT matrix and enforce bounded 8 MiB aggregate in-flight memory, >7 MiB/s/viewer at 100 ms RTT, and <200 ms completion spread. JSON fallback parity and live Android scrollback restoration remain explicit gaps.",
"commands": [
"pnpm exec vitest run --config config/vitest.config.ts src/main/runtime/rpc/terminal-subscribe-buffer.test.ts src/main/runtime/rpc/terminal-output-batching.test.ts src/main/runtime/rpc/terminal-multiplex.test.ts",
"pnpm exec vitest run --config config/vitest.config.ts src/main/runtime/rpc/terminal-subscribe-buffer.test.ts src/main/runtime/rpc/terminal-output-batching.test.ts src/main/runtime/rpc/terminal-multiplex.test.ts src/renderer/src/components/terminal-pane/pty-connection.test.ts src/renderer/src/components/terminal-pane/terminal-pty-ack-gate.test.ts src/renderer/src/lib/pane-manager/terminal-delivery-credit.test.ts src/renderer/src/runtime/remote-runtime-terminal-parse-backpressure.test.ts src/renderer/src/runtime/runtime-terminal-stream.test.ts --maxWorkers=1",
"ORCA_TERMINAL_PERF_BENCH=1 pnpm exec vitest run --config config/vitest.config.ts --disableConsoleIntercept src/main/runtime/rpc/terminal-multiplex-flow-control.bench.test.ts",
"pnpm --dir mobile exec vitest run --root .. mobile/src/session/mobile-native-chat-terminal-stream.test.ts",
"pnpm --dir mobile exec vitest run --root .. mobile/src/session/use-mobile-native-chat-terminal-stream.test.ts"
],
@@ -4625,6 +4627,12 @@
"src/main/runtime/rpc/terminal-subscribe-buffer.test.ts",
"src/main/runtime/rpc/terminal-output-batching.test.ts",
"src/main/runtime/rpc/terminal-multiplex.test.ts",
"src/main/runtime/rpc/terminal-multiplex-flow-control.bench.test.ts",
"src/renderer/src/components/terminal-pane/pty-connection.test.ts",
"src/renderer/src/components/terminal-pane/terminal-pty-ack-gate.test.ts",
"src/renderer/src/lib/pane-manager/terminal-delivery-credit.test.ts",
"src/renderer/src/runtime/remote-runtime-terminal-parse-backpressure.test.ts",
"src/renderer/src/runtime/runtime-terminal-stream.test.ts",
"mobile/src/session/mobile-native-chat-terminal-stream.test.ts",
"mobile/src/session/use-mobile-native-chat-terminal-stream.test.ts"
],
@@ -4649,7 +4657,33 @@
"assertions": [
"requested snapshots fall back smaller when serialized data exceeds the send budget",
"oversized live output frames are bounded for subscribed binary streams",
"multibyte live output flushes when encoded bytes reach the batch budget"
"multibyte live output flushes when encoded bytes reach the batch budget",
"adaptive credit grows only after ACK, stays globally bounded, and drains pending streams round-robin",
"send and recovery serialization failures detach once instead of leaking credit or retrying forever",
"32 active or pending slots cap aggregate queued output and repeated pending-slot subscribe cancels its older waiter"
]
},
{
"file": "src/renderer/src/lib/pane-manager/terminal-delivery-credit.test.ts",
"assertions": [
"nested synchronous deliveries restore the outer credit owner",
"unclaimed intentional discards settle automatically while every claimed scheduler child must settle before the parent credits"
]
},
{
"file": "src/renderer/src/runtime/remote-runtime-terminal-parse-backpressure.test.ts",
"assertions": [
"paired renderer ACK waits for xterm parse completion or explicit discard",
"192 KiB parsed output batches into one ACK while the 4 ms timer releases interactive output",
"malformed frames, malformed transformed output, disposal, late parse, renderer delivery failure, and ACK transport failure release credit or close the owning stream without reordering output"
]
},
{
"file": "src/main/runtime/rpc/terminal-multiplex-flow-control.bench.test.ts",
"assertions": [
"one through eight viewers stay within the 8 MiB aggregate adaptive window",
"the 100 ms RTT model sustains more than 7 MiB/s per viewer with less than 200 ms fairness spread",
"the opt-in benchmark reports RTT throughput, scheduler CPU time, exact protocol frame allocations, completion spread, and measured @xterm/headless parser CPU and retained heap"
]
},
{
@@ -4686,6 +4720,15 @@
"result": "passed",
"durationSeconds": 0.2,
"summary": "The focused mobile native-chat suite passed with 3 terminal-stream lifecycle assertions in the staged PR #5824 worktree."
},
{
"date": "2026-07-22",
"runner": "local",
"platform": "macos",
"command": "ORCA_TERMINAL_PERF_BENCH=1 pnpm exec vitest run --config config/vitest.config.ts --disableConsoleIntercept src/main/runtime/rpc/terminal-multiplex-flow-control.bench.test.ts",
"result": "passed",
"durationSeconds": 0.91,
"summary": "The 1/20/100 ms RTT x 1/4/8 viewer matrix stayed at or below 8 MiB in flight with zero completion spread. At 100 ms it modeled 18.8 MiB/s per viewer for 1-4 viewers and 9.7 MiB/s for 8 viewers. Measured @xterm/headless parsing was 26.7/63.6/95.3 aggregate MiB/s for 1/4/8 viewers, with 84.4/236.5/336.0 ms CPU and 2893/13409/28991 KiB retained heap for 4 MiB per viewer."
}
],
"runtimeBudget": {
@@ -4702,7 +4745,7 @@
},
"performanceBudget": {
"required": true,
"evidence": "This gate is the byte and batching budget for runtime/mobile terminal streaming."
"evidence": "Parsed/discarded credit uses 192 KiB/4 ms ACK batching, 512 KiB-to-2 MiB adaptive per-stream windows, a 2 MiB-to-8 MiB aggregate window, <=48 KiB output frames, <=256 KiB queued output per stream, and <=32 streams per connection (8 MiB aggregate pending output). The 64 MiB/viewer model gate requires >7 MiB/s/viewer at 100 ms RTT, <200 ms completion spread, and aggregate in-flight bytes <=8 MiB. The 2026-07-22 run modeled 9.7 MiB/s/viewer at 100 ms with eight viewers and measured @xterm/headless at 95.3 aggregate MiB/s, 336.0 ms parser CPU, and 28991 KiB retained heap for eight 4 MiB viewers."
},
"promotionCriteria": [
"Gate binary multiplex first.",
@@ -4711,7 +4754,8 @@
],
"knownGaps": [
"The pure mobile decision gate does not yet prove live Android WebView scrollback restore after a chat toggle.",
"Legacy JSON subscribe parity is undecided."
"Legacy JSON subscribe parity is undecided.",
"The parser measurement uses @xterm/headless; browser renderer/WebGL CPU, GPU, and allocation behavior still need packaged-app performance evidence."
],
"demotionRule": "Cannot promote while a supported stream path has uncapped snapshot or live-output buffering."
},