Failed warmups and repetitions were dropped from benchmark summaries, a failed startup, build or cleanup could leave an earlier success report in place, and a hung renderer or child process could block a run. Every attempt now keeps its status, metrics and error so summaries have the complete denominator; failures replace stale reports; browser and child-process lifecycles are bounded with cleanup of owned resources; and each criterion execution gets its own artifact directory. Co-authored-by: Mr0Bread <55754540+Mr0Bread@users.noreply.github.com>
5.5 KiB
IDE comparison
pnpm perf measures OpenIDEA from inside its webview. This measures any IDE
from the outside, the same way for each, so OpenIDEA can be compared with
IntelliJ IDEA (or anything else listed in apps.toml).
pip install -r perf/compare/requirements.txt
python perf/compare/bench.py ~/src/spring-petclinic --runs 10
python perf/compare/bench.py ~/src/spring-petclinic --runs 5 --wipe-caches # cold index
Each run launches the app on the project, waits until it is ready, idles, closes it, and records:
| Metric | How |
|---|---|
window_s |
first visible window whose title matches window_title (the project name) |
ready_s |
the process tree's CPU stays under --quiet-cores for --quiet-secs, after the window and any ready_after markers |
| markers | first/last time a log line matches, e.g. OpenIDEA indexed, IDEA exit dumb mode |
peak_rss_mb, rss_mb, uss_mb, pss_mb |
summed over every process the app started (WebView2/WebKit, jdtls, language servers, the JVM) |
total_cpu_s, idle_cpu_pct |
CPU time until ready; CPU while idle for --idle-secs |
exit_s |
until every process has exited |
cache_mb, install size |
cache_dirs, install_dir |
Apps run interleaved, rotating the order, after --warmup unaggregated runs.
Every attempt, including warmups and failed launches, is retained in
results/<time>-<project>-<unique>/: report.md, results.json, and each
launched run's stdout. Reports are refreshed before and after each attempt
and on errors or interruption; separate invocations never overwrite each other.
The table shows median (min–max) and valid/attempted measured-trial counts for every fixed metric, even when there are zero valid values. Only completely valid trials enter aggregates; failed trials retain partial measurements and explicit reasons in JSON and the attempts table. Warmups never enter medians but a failed warmup still fails the suite. An empty or incomplete suite is not successful.
Startup timeout, early exit/crash, observation/access errors, invalid or missing
required metrics, quit-command errors, forced termination, and interruption
produce a nonzero exit. JSON records status, reasons, warmup, and
metric_status (valid, missing, invalid, or unsupported). Nonfinite
numeric evidence is preserved as an invalid_number object, never JSON NaN.
Unsupported window observation, non-Linux PSS, and unconfigured cache directories
are explicit exclusions rather than missing required values. Unobserved optional
log markers do not fail a trial; ready_after markers are required.
--runs must be positive; --warmup and --cooldown may be zero. Other
durations and the CPU threshold must be finite and positive, and the quiet
period cannot exceed the startup timeout.
Shutdown always runs after a launch, including observation errors and Ctrl-C.
--close-timeout bounds the quit command plus graceful wait; each forced cleanup
has at most two additional seconds to reap processes. --command-timeout
(default 60 seconds) bounds --drop-os-cache, including interactive commands.
Only processes attributed to the launch are terminated; preexisting instances
are refused, never killed. Configured daemons remain explicitly excluded from
shutdown and forced cleanup. POSIX launch-owned process groups retain orphaned
children; on Windows discovery uses the recorded process ancestry.
Run the small process-fixture regressions without launching an installed IDE:
python -m unittest discover -s perf/compare -p 'test_bench.py'
Fair runs
- Release builds only; a running instance of the same executable (a
pnpm tauri devbuild too) would take over the launch, so the script refuses. - IDEA: open the project once by hand and trust it; turn off Reopen projects
on startup and Confirm before exiting; note the version,
-Xmxand whether Shared Indexes were on. - Same machine, power plan and plugged in; exclude the project and caches from antivirus scanning, or neither.
--drop-os-cacheruns a command before each launch for a cold OS cache:sync; echo 3 | sudo tee /proc/sys/vm/drop_caches(Linux),sudo purge(macOS),RAMMap64.exe -Et(Windows, admin).- Typing latency is not covered: measure it on screen with Typometer for both.
Platform notes
- Windows: windows via
user32, closed withWM_CLOSE. - macOS: windows via Quartz (
pyobjc-framework-Quartz; titles need Screen Recording permission). WebKit helpers are children of launchd and are attributed throughresponsibility_get_pid_responsible_for_pid. Without that API, unrelated WebKit processes are not guessed by name/start time or killed. - Linux: windows via
wmctrl(X11). Without itwindow_sis skipped and readiness counts from launch.