mirror of
https://github.com/stablyai/orca.git
synced 2026-10-16 16:03:26 +00:00
Prod verdict: v1.4.121-rc.0 bench — DSR-load p50 134->18.6ms (7.2x), agent-tui 2.0->11.2 MB/s, idle at Terminal.app parity; pipeline now cadence-bound
Co-authored-by: Orca <help@stably.ai>
This commit is contained in:
Jinwoo-H
Orca
@@ -479,6 +479,36 @@ incl. stream-opcode collision (chain `Ack=12` vs main's #7205-era
|
||||
gate×drain risk; (3) cadence tuning toward the 10× line; (4) rig
|
||||
extensions + P90 telemetry (tasks #3/#8).
|
||||
|
||||
### 2026-07-03 — PROD VERDICT: v1.4.121-rc.0 benchmarked (the headline numbers)
|
||||
|
||||
Same rig, same protocol, same machine as the 1.4.91 baseline:
|
||||
|
||||
| metric | 1.4.91 baseline | v1.4.121-rc.0 | change |
|
||||
|---|---|---|---|
|
||||
| DSR idle p50 | 0.69 ms | **0.44 ms** | = Terminal.app (0.45) |
|
||||
| DSR under load p50 | 134 ms | **18.6 ms** | 7.2x |
|
||||
| DSR under load p99 | 292 ms | **29.7 ms** | 9.8x |
|
||||
| agent-tui | 2.0 MB/s | **11.2 MB/s** | 5.6x |
|
||||
| styles-stress | 7.8 MB/s | **10.4 MB/s** | 1.3x |
|
||||
| ascii-log | 13 MB/s | 11.0 MB/s | ~0.85x |
|
||||
| cjk-emoji | 15 MB/s | 12.2 MB/s | ~0.81x |
|
||||
|
||||
Reading: the anomalous TUI penalty is GONE — all four fixtures now sit at
|
||||
a uniform ~11-12 MB/s, which is the scheduler pacing ceiling, not parse
|
||||
CPU (prod ≈ dev for both latency and throughput; the pipeline is
|
||||
cadence-bound, so faster prod code changes nothing). That uniform cap
|
||||
also explains plain-text dipping slightly below baseline: ascii/cjk used
|
||||
to run unpaced ahead of the old scheduler; now everything flows through
|
||||
the same parse-clocked path. Goal line check: 18.6 ms = 41x Terminal.app
|
||||
under load (goal was 10x = 4.5 ms) — NOT met; down from 300x. Idle IS at
|
||||
parity. The remaining 4x is the named cadence stack (daemon 8 ms batch,
|
||||
scheduler drain ticks + 8x16KB per-tick budget, xterm 12 ms slices) —
|
||||
next lever, tunable, tracked as follow-up. p99 tail (the freeze class)
|
||||
is 29.7 ms — users cannot perceive it.
|
||||
|
||||
Caveat: measured on the user's live app (this session active in it);
|
||||
idle p99 118 ms reflects that activity, not the terminal path.
|
||||
|
||||
## Success criteria (baseline-relative; finalize after task 1)
|
||||
|
||||
- DSR-under-load p90 in Orca within striking distance of iTerm2 on the same
|
||||
|
||||
+116
@@ -0,0 +1,116 @@
|
||||
{
|
||||
"benchmark": "terminal-pipeline",
|
||||
"label": "prod-rc-1.4.121",
|
||||
"timestamp": "2026-07-03T07:50:57.563Z",
|
||||
"terminal": {
|
||||
"program": "Orca",
|
||||
"version": "1.4.121-rc.0",
|
||||
"term": "xterm-256color",
|
||||
"cols": 114,
|
||||
"rows": 85,
|
||||
"tty": true
|
||||
},
|
||||
"machine": {
|
||||
"platform": "darwin",
|
||||
"arch": "arm64",
|
||||
"cpu": "Apple M4",
|
||||
"release": "25.2.0",
|
||||
"node": "v26.0.0"
|
||||
},
|
||||
"config": {
|
||||
"sizeMb": 10,
|
||||
"iterations": 5,
|
||||
"dsrCount": 200
|
||||
},
|
||||
"dsrIdle": {
|
||||
"count": 200,
|
||||
"p50": 0.44337500000006,
|
||||
"p90": 1.1750409999999647,
|
||||
"p99": 118.32008399999995,
|
||||
"max": 132.676959,
|
||||
"timeouts": 0
|
||||
},
|
||||
"throughput": [
|
||||
{
|
||||
"fixture": "ascii-log",
|
||||
"fixtureBytes": 10485767,
|
||||
"wallMs": {
|
||||
"count": 5,
|
||||
"p50": 907.2806660000001,
|
||||
"p90": 974.0933340000001,
|
||||
"p99": 974.0933340000001,
|
||||
"max": 974.0933340000001
|
||||
},
|
||||
"mbPerSec": {
|
||||
"count": 5,
|
||||
"p50": 11.021955002973923,
|
||||
"p90": 12.697340893816287,
|
||||
"p99": 12.697340893816287,
|
||||
"max": 12.697340893816287
|
||||
}
|
||||
},
|
||||
{
|
||||
"fixture": "cjk-emoji",
|
||||
"fixtureBytes": 10485782,
|
||||
"wallMs": {
|
||||
"count": 5,
|
||||
"p50": 821.5044170000001,
|
||||
"p90": 922.7275409999984,
|
||||
"p99": 922.7275409999984,
|
||||
"max": 922.7275409999984
|
||||
},
|
||||
"mbPerSec": {
|
||||
"count": 5,
|
||||
"p50": 12.172814623874334,
|
||||
"p90": 15.028801300957284,
|
||||
"p99": 15.028801300957284,
|
||||
"max": 15.028801300957284
|
||||
}
|
||||
},
|
||||
{
|
||||
"fixture": "agent-tui",
|
||||
"fixtureBytes": 10486459,
|
||||
"wallMs": {
|
||||
"count": 5,
|
||||
"p50": 895.6726660000004,
|
||||
"p90": 939.9889590000002,
|
||||
"p99": 939.9889590000002,
|
||||
"max": 939.9889590000002
|
||||
},
|
||||
"mbPerSec": {
|
||||
"count": 5,
|
||||
"p50": 11.16553736423521,
|
||||
"p90": 11.476432557326321,
|
||||
"p99": 11.476432557326321,
|
||||
"max": 11.476432557326321
|
||||
}
|
||||
},
|
||||
{
|
||||
"fixture": "styles-stress",
|
||||
"fixtureBytes": 10485774,
|
||||
"wallMs": {
|
||||
"count": 5,
|
||||
"p50": 962.8965839999983,
|
||||
"p90": 1023.915500000001,
|
||||
"p99": 1023.915500000001,
|
||||
"max": 1023.915500000001
|
||||
},
|
||||
"mbPerSec": {
|
||||
"count": 5,
|
||||
"p50": 10.385345132183424,
|
||||
"p90": 10.678530762733823,
|
||||
"p99": 10.678530762733823,
|
||||
"max": 10.678530762733823
|
||||
}
|
||||
}
|
||||
],
|
||||
"dsrUnderLoad": {
|
||||
"paceBytesPerSec": 1048576,
|
||||
"count": 50,
|
||||
"p50": 18.635999999998603,
|
||||
"p90": 23.42462499999965,
|
||||
"p99": 29.698333000000275,
|
||||
"max": 29.698333000000275,
|
||||
"timeouts": 0
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user