mirror of
https://github.com/l0ng-ai/tty7.git
synced 2026-09-21 16:02:20 +00:00
l0ng-aiandl0ng-ai
21b7f82392
* feat(completion): execute dynamic generators for live candidates The completion engine consumed Fig specs' static shape but never ran their dynamic generators, so positions whose candidates come from the live system — ssh hosts, git branches — fell through to filesystem path completion (#51: ssh <Tab> listed the cwd). Local-only by design: the pure engine returns each pending script, the view runs it on the background executor (/bin/sh -c in the session cwd, 800ms timeout, kill-on-drop, 256KiB stdout cap, 5s TTL cache) and merges the parsed lines into the open menu, generation-tagged so a result can't outlive its session. A per-script parser registry ports the specs' dropped postProcess transforms (git markers, docker {{json .}}, package.json scripts, …); unmatched scripts default to one-candidate-per-line, and hopeless outputs are suppressed rather than inserted as garbage. ssh/scp/sftp/rsync specs gain host generators reading ~/.ssh/config (Include-aware, wildcard patterns skipped) and known_hosts (hashed entries skipped, [host]:port unwrapped). Fixes #51 Co-authored-by: l0ng-ai <24760907+l0ng-ai@users.noreply.github.com> * docs(comments): describe borrowed conventions directly Prior-art name-drops in comments aged poorly as the implementations diverged; keep the behavioral rationale, drop the product citations. * fix(completion): make generator execution CI-portable Linux: sh -c may fork the command instead of exec'ing it, so killing only the shell on timeout left a grandchild holding the stdout pipe — the reader (and the caller) then blocked until the grandchild exited on its own. Spawn the child as its own process-group leader and kill the group; the timeout test now forces the fork case (trailing true) so the group-kill is what's actually proven. Windows: generator scripts are POSIX sh + awk, so the execution path now compiles to no-suggestions there instead of failing at runtime on a missing /bin/sh; the process-spawning tests are Unix-only to match. --------- Co-authored-by: l0ng-ai <24760907+l0ng-ai@users.noreply.github.com>
142 lines
7.2 KiB
Markdown
142 lines
7.2 KiB
Markdown
# Terminal benchmark harness
|
||
|
||
Reproducible throughput/FPS/memory benchmarks for tty7 against Alacritty,
|
||
Ghostty and Kitty, following the methodology of
|
||
[moktavizen/terminal-benchmark](https://github.com/moktavizen/terminal-benchmark).
|
||
macOS only (drivers use `/Applications` paths, BSD `ps`, `pkill`).
|
||
|
||
## What it measures
|
||
|
||
| Test | Method | Collects |
|
||
| --- | --- | --- |
|
||
| **Plaintext IO** | `cat` an 11 MB text file inside the terminal, 5 runs | elapsed ms per run (lower = better) |
|
||
| **Frame rate** | [DOOM-fire-zig](https://github.com/const-void/DOOM-fire-zig), 5 runs × ~14 s | cumulative average fps per run (higher = better) |
|
||
| **Memory** | cold launch, default shell, idle 6 s | RSS in KB (tty7 = GUI + daemon) |
|
||
|
||
The methodology source also measures **input latency**, but with a high-speed
|
||
camera ([is-it-snappy](https://github.com/chadaustin/is-it-snappy)) — that
|
||
can't be automated here and is skipped.
|
||
|
||
## Usage
|
||
|
||
```bash
|
||
scripts/bench/setup.sh # once: corpus + patched DOOM-fire (+ zig 0.14 if needed)
|
||
cargo build --release # the tty7 under test
|
||
|
||
scripts/bench/run_one.sh tty7 io # ALWAYS first: records the reference grid
|
||
scripts/bench/run_one.sh alacritty io # matches tty7's grid automatically
|
||
scripts/bench/run_one.sh ghostty io
|
||
scripts/bench/run_one.sh kitty io
|
||
scripts/bench/run_one.sh tty7 fire # same order for the fire test
|
||
scripts/bench/run_one.sh alacritty fire
|
||
scripts/bench/run_one.sh ghostty fire
|
||
scripts/bench/run_one.sh kitty fire
|
||
scripts/bench/mem.sh # all four terminals, 3 runs each
|
||
```
|
||
|
||
Results land in `.bench/results/` (override the work dir with
|
||
`$TTY7_BENCH_DIR`, the tty7 binary with `$TTY7_BIN`).
|
||
|
||
**While a run is up: don't type into the window, and don't hide or fully
|
||
occlude it** — macOS throttles occluded windows and the FPS numbers collapse
|
||
(observed: ~950 fps → ~360 fps when the window was hidden mid-run).
|
||
|
||
## How it drives each terminal
|
||
|
||
Every terminal opens a real window whose *shell* is the benchmark script, so
|
||
the measurement includes the full input path the user experiences:
|
||
|
||
- **tty7**: an isolated `--config-dir` under the work dir whose `config.json`
|
||
sets `shell` to the script. GUI + daemon are launched fresh and killed
|
||
(by config-dir-scoped `pkill`) after the run — a daily-driver tty7 and its
|
||
daemon are never touched.
|
||
- **Alacritty**: `--config-file <empty>` (isolates the user's config) plus
|
||
`-o window.dimensions.…` for the grid, `-e` for the script.
|
||
- **Ghostty**: `--config-default-files=false` plus `--window-width/height`
|
||
(cells) and `-e`. `--window-save-state=never` matters: macOS window
|
||
restoration otherwise overrides the requested size.
|
||
- **Kitty**: `--config NONE` plus `-o initial_window_width/height=<n>c` (the
|
||
`c` suffix means cells); the script is passed as trailing args (no `-e`).
|
||
`-o remember_window_size=no` matters: it defaults to yes even under
|
||
`--config NONE`, and the restored size overrides `initial_window_*`.
|
||
|
||
iTerm2/Terminal.app are not benchmarked: they would need AppleScript driving
|
||
and can't cleanly isolate config.
|
||
|
||
Grid fairness: tty7 has no size flag, so its default window is the reference —
|
||
run tty7 first, and the driver reads the recorded `grid:` line to size the
|
||
other terminals identically (cells, not pixels; fonts differ).
|
||
|
||
## Why DOOM-fire is patched
|
||
|
||
`doom-fire-fps.patch` (applied by `setup.sh`) makes DOOM-fire dump its
|
||
cumulative average fps to `$DOOM_FPS_FILE` every 30 frames. The upstream
|
||
binary only *paints* the number, and recording the output stream to parse it
|
||
back (`script(1)`) is a trap: at 500+ fps the recording grows to gigabytes in
|
||
seconds and its disk writes throttle later runs by 4-8×. The fps definition is
|
||
unchanged — total frames / elapsed seconds since the fire started, the same
|
||
number painted on screen.
|
||
|
||
## Recorded baseline (2026-07-03)
|
||
|
||
Apple M1 Pro, 32 GB, macOS 26.3.1, grid 155×40, release builds, defaults.
|
||
Optimization = the `VecDeque` replay ring + coalesced `Output` frames +
|
||
backpressure gate (see CHANGELOG "Terminal throughput ~12× faster").
|
||
|
||
| Test | tty7 (before) | tty7 (after) | Alacritty | Ghostty | Kitty |
|
||
| --- | ---: | ---: | ---: | ---: | ---: |
|
||
| Plaintext IO, 5-run avg | 2030 ms | **161 ms** | 232 ms | 183 ms | 217 ms |
|
||
| DOOM-fire, 5-run avg | 47 fps | **920 fps** | 542 fps | 533 fps | 546 fps |
|
||
| Memory (GUI+daemon) | 100 MB | **105 MB** | 86 MB | 112 MB | — |
|
||
|
||
Kitty (0.47.4) was measured the same day at the same 155×40 grid; its memory
|
||
run was skipped. Upstream's Kitty frame-rate dominance (Linux/Wayland) does
|
||
not reproduce on macOS — here it lands in the same band as Alacritty/Ghostty.
|
||
|
||
Diagnosis notes for posterity: macOS PTYs deliver ~1 KiB per read. Before the
|
||
fix, every read into a full 8 MiB `Vec` ring memmoved the whole ring
|
||
(`drain(..overflow)`), eating ~92% of the daemon reader's time — visible as
|
||
"run 1 fast, later runs slow" (the ring fills during run 1). `TTY7_TRACE=1`
|
||
on both the GUI and a foreground daemon prints the per-second accounting that
|
||
localized this.
|
||
|
||
## Recorded baseline (2026-07-04, second throughput pass)
|
||
|
||
Same machine and grid, all four terminals re-run the same day. Optimization =
|
||
the CHANGELOG "second throughput pass" batch (16 MiB gate, 256 KiB socket
|
||
buffers, client-side Output batching, memchr OSC fast paths, atomic gate,
|
||
QoS promotion).
|
||
|
||
All four terminals measured back-to-back in one quiet-machine session:
|
||
|
||
| Test | tty7 (before) | tty7 (after) | Alacritty | Ghostty | Kitty |
|
||
| --- | ---: | ---: | ---: | ---: | ---: |
|
||
| Plaintext IO, 5-run avg | 154 ms | **95 ms** | 239 ms | 179 ms | 185 ms |
|
||
| DOOM-fire, 5-run avg | ~760 fps¹ | **888 fps** | 485 fps | 552 fps | 616 fps |
|
||
| Memory (GUI+daemon) | 112 MB | 115 MB | 105 MB | 128 MB | 130 MB |
|
||
|
||
¹ The tty7-before numbers were measured while the machine was busy (builds +
|
||
tracing running alongside); on the later quiet machine the same pre-pass
|
||
pipeline would have landed near its 07-03 920 fps. The fire before/after
|
||
delta is therefore mostly ambient load, not the optimization — see the notes
|
||
below. After the pass, quiet-machine fire runs tightened to 882–894 (±0.7%).
|
||
|
||
Notes for interpreting these numbers, learned the hard way:
|
||
|
||
- **The day's fps band matters more than the run.** The same pre-pass binary
|
||
that recorded 920 fps on 07-03 measured 728–857 on 07-04; competitors
|
||
reproduced within ±3%. tty7's fire number is drain-rate-bound and therefore
|
||
sensitive to ambient machine load in a way the (slower) competitors aren't.
|
||
Only compare tty7-vs-tty7 fire numbers from the same session.
|
||
- **DOOM-fire is producer-bound, not terminal-bound, at this level.** Under a
|
||
raw do-nothing PTY reader it produces ~96 MB/s at a constant ~87 KB/frame —
|
||
i.e. ~1050–1100 fps is the machine's ceiling for *any* terminal, and fps
|
||
scales linearly with drain rate (capped-drain probe: 95 MB/s → 1092 fps,
|
||
60 MB/s → 690 fps). tty7's steady seconds already drain at 93–98 MB/s; the
|
||
gap to the ceiling is whole seconds where the *producer* gets descheduled.
|
||
- **`cat` completion time is a drain benchmark, not a render benchmark.** The
|
||
16 MiB gate lets an 11 MB burst leave the PTY at device speed while the
|
||
client parses behind; sustained plaintext drain is 148 MB/s against a
|
||
~170 MB/s raw-reader ceiling (the client's VT parser, ~0.7 core, is the
|
||
remaining sustained-throughput limit).
|