VS Code head-to-head: Orca beats/ties 5 of 6 metrics (16x idle, 5x styles-stress, better p99); load p50 gap attributed to ACK window + timer-clamped drain cadence

Co-authored-by: Orca <help@stably.ai>
2026-09-26 08:02:38 +00:00 · 2026-07-03 16:20:28 -04:00
parent a03918156d
commit 4252a7eab4
2 changed files with 144 additions and 0 deletions
@@ -509,6 +509,34 @@ is 29.7 ms — users cannot perceive it.
 Caveat: measured on the user's live app (this session active in it);
 idle p99 118 ms reflects that activity, not the terminal path.

+### 2026-07-03 — Same-engine reference: VS Code head-to-head (same machine, same rig)
+
+| metric | Orca v1.4.121-rc.0 | VS Code | verdict |
+|---|---|---|---|
+| DSR idle p50 | **0.44 ms** | 7.00 ms | Orca 16x faster |
+| DSR load p50 | 18.6 ms | **7.18 ms** | VS Code 2.6x faster |
+| DSR load p99 | **29.7 ms** | 43.4 ms | Orca 1.5x better tail |
+| ascii-log | **11.0 MB/s** | 9.0 | Orca +22% |
+| cjk-emoji | 12.2 | 11.3 | tie |
+| agent-tui | 11.2 | 11.7 | tie |
+| styles-stress | **10.4 MB/s** | 2.0 | Orca 5.2x |
+
+Orca now beats or ties the best-known xterm.js terminal on 5 of 6
+metrics — including 16x at idle (what users feel all day) and 5x on
+SGR-heavy output — and holds a better p99 tail under load. Throughput
+sits at the shared engine ceiling (~9-12 MB/s), confirming the class
+limit.
+
+The one loss (load p50) has a clean mechanism: VS Code's producer flow
+control caps unacked output at ~100KB, so its standing queue is
+~100KB / 11.7 MB/s ≈ 8.5 ms — matching its 7.18. Our standing queue
+(18.6 ms ≈ ~200KB at 11 MB/s) is set by the main→renderer ACK window
+(512KB/pty high water) + drain re-arm cadence (Chromium clamps nested
+setTimeout to ~4ms). Two levers, both cheap to test: (1) MessageChannel
+drain scheduling (sub-ms re-arm; also raises the throughput ceiling);
+(2) tighter effective in-flight window on the renderer delivery path.
+Target: VS Code's ~7ms class or below without giving back throughput.
+
 ## Success criteria (baseline-relative; finalize after task 1)

 - DSR-under-load p90 in Orca within striking distance of iTerm2 on the same
@@ -0,0 +1,116 @@
+{
+  "benchmark": "terminal-pipeline",
+  "label": "vscode-1.4.121-era",
+  "timestamp": "2026-07-03T20:17:14.495Z",
+  "terminal": {
+    "program": "vscode",
+    "version": "3.9.16",
+    "term": "xterm-256color",
+    "cols": 91,
+    "rows": 31,
+    "tty": true
+  },
+  "machine": {
+    "platform": "darwin",
+    "arch": "arm64",
+    "cpu": "Apple M4",
+    "release": "25.2.0",
+    "node": "v26.0.0"
+  },
+  "config": {
+    "sizeMb": 10,
+    "iterations": 5,
+    "dsrCount": 200
+  },
+  "dsrIdle": {
+    "count": 200,
+    "p50": 7.004625000000033,
+    "p90": 7.78366699999998,
+    "p99": 14.745250000000397,
+    "max": 16.20970800000009,
+    "timeouts": 0
+  },
+  "throughput": [
+    {
+      "fixture": "ascii-log",
+      "fixtureBytes": 10485767,
+      "wallMs": {
+        "count": 5,
+        "p50": 1114.0425409999998,
+        "p90": 1135.616083,
+        "p99": 1135.616083,
+        "max": 1135.616083
+      },
+      "mbPerSec": {
+        "count": 5,
+        "p50": 8.976323890417948,
+        "p90": 9.17852838524113,
+        "p99": 9.17852838524113,
+        "max": 9.17852838524113
+      }
+    },
+    {
+      "fixture": "cjk-emoji",
+      "fixtureBytes": 10485782,
+      "wallMs": {
+        "count": 5,
+        "p50": 884.9254999999994,
+        "p90": 936.1878749999996,
+        "p99": 936.1878749999996,
+        "max": 936.1878749999996
+      },
+      "mbPerSec": {
+        "count": 5,
+        "p50": 11.300410012859803,
+        "p90": 13.248065106709598,
+        "p99": 13.248065106709598,
+        "max": 13.248065106709598
+      }
+    },
+    {
+      "fixture": "agent-tui",
+      "fixtureBytes": 10486010,
+      "wallMs": {
+        "count": 5,
+        "p50": 851.7975839999999,
+        "p90": 882.9034170000014,
+        "p99": 882.9034170000014,
+        "max": 882.9034170000014
+      },
+      "mbPerSec": {
+        "count": 5,
+        "p50": 11.74015823292016,
+        "p90": 11.934587392159806,
+        "p99": 11.934587392159806,
+        "max": 11.934587392159806
+      }
+    },
+    {
+      "fixture": "styles-stress",
+      "fixtureBytes": 10485774,
+      "wallMs": {
+        "count": 5,
+        "p50": 4919.174958,
+        "p90": 5186.167792,
+        "p99": 5186.167792,
+        "max": 5186.167792
+      },
+      "mbPerSec": {
+        "count": 5,
+        "p50": 2.0328639328384774,
+        "p90": 2.0547078806834653,
+        "p99": 2.0547078806834653,
+        "max": 2.0547078806834653
+      }
+    }
+  ],
+  "dsrUnderLoad": {
+    "paceBytesPerSec": 1048576,
+    "count": 50,
+    "p50": 7.176874999997381,
+    "p90": 11.226500000004307,
+    "p99": 43.379000000000815,
+    "max": 43.379000000000815,
+    "timeouts": 0
+  }
+}