mirror of
https://github.com/stablyai/orca.git
synced 2026-10-19 00:03:20 +00:00
test(reliability): benchmark terminal flow control
This commit is contained in:
@@ -4608,16 +4608,18 @@
|
||||
"macos"
|
||||
],
|
||||
"coveredProviders": [],
|
||||
"coverageNotes": "Local macOS evidence over the runtime-RPC stream budgets on main@1282f5c2d. PR #5824 adds a platform-neutral mobile decision gate proving chat-covered terminal streams pause and resume only after the mounted WebView is ready; live Android restore evidence remains required. The pending stack adds byte-exact 512KB/2MB/256KB/48KB budget assertions; legacy JSON subscribe parity remains undecided.",
|
||||
"coverageNotes": "Local macOS evidence covers runtime-RPC stream budgets plus paired-renderer parse/discard credit. Deferred credit is shared by local and remote transports, batches ACKs at 192 KiB or 4 ms, grows per-stream windows from 512 KiB to 2 MiB and aggregate windows from 2 MiB to 8 MiB, bounds queued output to 256 KiB per stream, and caps each multiplex connection at 32 active or pending streams for an 8 MiB aggregate pending-output ceiling. Deterministic tests cover replay ordering, stale generations, malformed frames, hidden panes, queue eviction, disposal, send/recovery failure, repeated pending-slot replacement, reconnect, and round-robin fairness. The opt-in benchmark covers 1/20/100 ms RTT and 1/4/8 viewers, exact protocol-frame allocations, scheduler CPU, and measured @xterm/headless parser CPU/retained heap. Live Android restore evidence, browser/WebGL parser measurements, and legacy JSON subscribe parity remain required.",
|
||||
"motivatingLinks": [
|
||||
"https://github.com/stablyai/orca/pull/6951",
|
||||
"https://github.com/stablyai/orca/pull/6955",
|
||||
"https://github.com/stablyai/orca/pull/7009"
|
||||
],
|
||||
"invariant": "Runtime and mobile terminal subscriptions must cap initial snapshots, live output buffered while snapshots load, chunk sizes, and batch sizes; a terminal covered by native chat must have no live output subscription and must restore from fresh scrollback when revealed, while preserving output order, input locks, resize/driver events, and fallback parity or explicit fallback deprecation.",
|
||||
"oracle": "The current executable slice asserts mobile initial snapshots downgrade until they fit <=512KB, requested binary snapshots downgrade until they fit <=2MB, binary live output queued while the initial snapshot loads stays <=256KB while preserving the newest tail, large binary output is split into <=48KB frames, output bursts are coalesced before emit, aborts do not register stale listeners, and stale mobile resize re-stream completions are dropped. The mobile native-chat decision test asserts an active stream pauses while covered and resumes only for a ready active terminal. JSON fallback parity and live Android scrollback restoration remain explicit gaps.",
|
||||
"invariant": "Runtime and mobile terminal subscriptions must cap initial snapshots, live output buffered while snapshots load, chunk sizes, batches, and aggregate in-flight credit. ACK means the renderer parsed the bytes or intentionally discarded them; receipt-time ACK is forbidden. Every replay, stale-generation, malformed-frame, hidden-pane, eviction, disposal, error, and reconnect path must settle credit exactly once so streams neither leak memory nor stall. A terminal covered by native chat must restore from fresh scrollback when revealed, while preserving output order, input locks, resize/driver events, fairness, and fallback parity or explicit fallback deprecation.",
|
||||
"oracle": "Assert mobile initial snapshots downgrade until they fit <=512KB, requested binary snapshots downgrade until they fit <=2MB, live output queued while snapshots load stays <=256KB per stream, large output splits into <=48KB frames, and output bursts coalesce. Feed paired output through the xterm parse callback and prove ACK is deferred until parse or intentional discard, then inject stale generation, malformed/transformed frames, replay failure, queue eviction, hidden panes, pane disposal, ACK send failure, recovery serialization failure, and reconnect races; assert ordered replay and exactly-once credit settlement. Fill the aggregate window across bulk and interactive streams, ACK once, and prove round-robin progress. Run the opt-in 64 MiB/viewer RTT matrix and enforce bounded 8 MiB aggregate in-flight memory, >7 MiB/s/viewer at 100 ms RTT, and <200 ms completion spread. JSON fallback parity and live Android scrollback restoration remain explicit gaps.",
|
||||
"commands": [
|
||||
"pnpm exec vitest run --config config/vitest.config.ts src/main/runtime/rpc/terminal-subscribe-buffer.test.ts src/main/runtime/rpc/terminal-output-batching.test.ts src/main/runtime/rpc/terminal-multiplex.test.ts",
|
||||
"pnpm exec vitest run --config config/vitest.config.ts src/main/runtime/rpc/terminal-subscribe-buffer.test.ts src/main/runtime/rpc/terminal-output-batching.test.ts src/main/runtime/rpc/terminal-multiplex.test.ts src/renderer/src/components/terminal-pane/pty-connection.test.ts src/renderer/src/components/terminal-pane/terminal-pty-ack-gate.test.ts src/renderer/src/lib/pane-manager/terminal-delivery-credit.test.ts src/renderer/src/runtime/remote-runtime-terminal-parse-backpressure.test.ts src/renderer/src/runtime/runtime-terminal-stream.test.ts --maxWorkers=1",
|
||||
"ORCA_TERMINAL_PERF_BENCH=1 pnpm exec vitest run --config config/vitest.config.ts --disableConsoleIntercept src/main/runtime/rpc/terminal-multiplex-flow-control.bench.test.ts",
|
||||
"pnpm --dir mobile exec vitest run --root .. mobile/src/session/mobile-native-chat-terminal-stream.test.ts",
|
||||
"pnpm --dir mobile exec vitest run --root .. mobile/src/session/use-mobile-native-chat-terminal-stream.test.ts"
|
||||
],
|
||||
@@ -4625,6 +4627,12 @@
|
||||
"src/main/runtime/rpc/terminal-subscribe-buffer.test.ts",
|
||||
"src/main/runtime/rpc/terminal-output-batching.test.ts",
|
||||
"src/main/runtime/rpc/terminal-multiplex.test.ts",
|
||||
"src/main/runtime/rpc/terminal-multiplex-flow-control.bench.test.ts",
|
||||
"src/renderer/src/components/terminal-pane/pty-connection.test.ts",
|
||||
"src/renderer/src/components/terminal-pane/terminal-pty-ack-gate.test.ts",
|
||||
"src/renderer/src/lib/pane-manager/terminal-delivery-credit.test.ts",
|
||||
"src/renderer/src/runtime/remote-runtime-terminal-parse-backpressure.test.ts",
|
||||
"src/renderer/src/runtime/runtime-terminal-stream.test.ts",
|
||||
"mobile/src/session/mobile-native-chat-terminal-stream.test.ts",
|
||||
"mobile/src/session/use-mobile-native-chat-terminal-stream.test.ts"
|
||||
],
|
||||
@@ -4649,7 +4657,33 @@
|
||||
"assertions": [
|
||||
"requested snapshots fall back smaller when serialized data exceeds the send budget",
|
||||
"oversized live output frames are bounded for subscribed binary streams",
|
||||
"multibyte live output flushes when encoded bytes reach the batch budget"
|
||||
"multibyte live output flushes when encoded bytes reach the batch budget",
|
||||
"adaptive credit grows only after ACK, stays globally bounded, and drains pending streams round-robin",
|
||||
"send and recovery serialization failures detach once instead of leaking credit or retrying forever",
|
||||
"32 active or pending slots cap aggregate queued output and repeated pending-slot subscribe cancels its older waiter"
|
||||
]
|
||||
},
|
||||
{
|
||||
"file": "src/renderer/src/lib/pane-manager/terminal-delivery-credit.test.ts",
|
||||
"assertions": [
|
||||
"nested synchronous deliveries restore the outer credit owner",
|
||||
"unclaimed intentional discards settle automatically while every claimed scheduler child must settle before the parent credits"
|
||||
]
|
||||
},
|
||||
{
|
||||
"file": "src/renderer/src/runtime/remote-runtime-terminal-parse-backpressure.test.ts",
|
||||
"assertions": [
|
||||
"paired renderer ACK waits for xterm parse completion or explicit discard",
|
||||
"192 KiB parsed output batches into one ACK while the 4 ms timer releases interactive output",
|
||||
"malformed frames, malformed transformed output, disposal, late parse, renderer delivery failure, and ACK transport failure release credit or close the owning stream without reordering output"
|
||||
]
|
||||
},
|
||||
{
|
||||
"file": "src/main/runtime/rpc/terminal-multiplex-flow-control.bench.test.ts",
|
||||
"assertions": [
|
||||
"one through eight viewers stay within the 8 MiB aggregate adaptive window",
|
||||
"the 100 ms RTT model sustains more than 7 MiB/s per viewer with less than 200 ms fairness spread",
|
||||
"the opt-in benchmark reports RTT throughput, scheduler CPU time, exact protocol frame allocations, completion spread, and measured @xterm/headless parser CPU and retained heap"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -4686,6 +4720,15 @@
|
||||
"result": "passed",
|
||||
"durationSeconds": 0.2,
|
||||
"summary": "The focused mobile native-chat suite passed with 3 terminal-stream lifecycle assertions in the staged PR #5824 worktree."
|
||||
},
|
||||
{
|
||||
"date": "2026-07-22",
|
||||
"runner": "local",
|
||||
"platform": "macos",
|
||||
"command": "ORCA_TERMINAL_PERF_BENCH=1 pnpm exec vitest run --config config/vitest.config.ts --disableConsoleIntercept src/main/runtime/rpc/terminal-multiplex-flow-control.bench.test.ts",
|
||||
"result": "passed",
|
||||
"durationSeconds": 0.91,
|
||||
"summary": "The 1/20/100 ms RTT x 1/4/8 viewer matrix stayed at or below 8 MiB in flight with zero completion spread. At 100 ms it modeled 18.8 MiB/s per viewer for 1-4 viewers and 9.7 MiB/s for 8 viewers. Measured @xterm/headless parsing was 26.7/63.6/95.3 aggregate MiB/s for 1/4/8 viewers, with 84.4/236.5/336.0 ms CPU and 2893/13409/28991 KiB retained heap for 4 MiB per viewer."
|
||||
}
|
||||
],
|
||||
"runtimeBudget": {
|
||||
@@ -4702,7 +4745,7 @@
|
||||
},
|
||||
"performanceBudget": {
|
||||
"required": true,
|
||||
"evidence": "This gate is the byte and batching budget for runtime/mobile terminal streaming."
|
||||
"evidence": "Parsed/discarded credit uses 192 KiB/4 ms ACK batching, 512 KiB-to-2 MiB adaptive per-stream windows, a 2 MiB-to-8 MiB aggregate window, <=48 KiB output frames, <=256 KiB queued output per stream, and <=32 streams per connection (8 MiB aggregate pending output). The 64 MiB/viewer model gate requires >7 MiB/s/viewer at 100 ms RTT, <200 ms completion spread, and aggregate in-flight bytes <=8 MiB. The 2026-07-22 run modeled 9.7 MiB/s/viewer at 100 ms with eight viewers and measured @xterm/headless at 95.3 aggregate MiB/s, 336.0 ms parser CPU, and 28991 KiB retained heap for eight 4 MiB viewers."
|
||||
},
|
||||
"promotionCriteria": [
|
||||
"Gate binary multiplex first.",
|
||||
@@ -4711,7 +4754,8 @@
|
||||
],
|
||||
"knownGaps": [
|
||||
"The pure mobile decision gate does not yet prove live Android WebView scrollback restore after a chat toggle.",
|
||||
"Legacy JSON subscribe parity is undecided."
|
||||
"Legacy JSON subscribe parity is undecided.",
|
||||
"The parser measurement uses @xterm/headless; browser renderer/WebGL CPU, GPU, and allocation behavior still need packaged-app performance evidence."
|
||||
],
|
||||
"demotionRule": "Cannot promote while a supported stream path has uncapped snapshot or live-output buffering."
|
||||
},
|
||||
|
||||
Reference in New Issue
Block a user