mirror of
https://github.com/neondatabase/neon.git
synced 2026-08-18 20:18:23 +00:00
Fedor Dikarev
9a6ace9bde
## Problem Benchmarks results are inconsistent on existing small-metal runners ## Summary of changes Introduce new `unit-perf` runners, and lets run benchmark on them. The new hardware has slower, but consistent, CPU frequency - if run with default governor schedutil. Thus we needed to adjust some testcases' timeouts and add some retry steps where hard-coded timeouts couldn't be increased without changing the system under test. - [wait_for_last_record_lsn](https://github.com/neondatabase/neon/blob/6592d69a6700a2bd2e9f60c22af138ea0dafbdd0/test_runner/fixtures/pageserver/utils.py#L193) 1000s -> 2000s - [test_branch_creation_many](https://github.com/neondatabase/neon/pull/11409/files#diff-2ebfe76f89004d563c7e53e3ca82462e1d85e92e6d5588e8e8f598bbe119e927) 1000s - [test_ingest_insert_bulk](https://github.com/neondatabase/neon/pull/11409/files#diff-e90e685be4a87053bc264a68740969e6a8872c8897b8b748d0e8c5f683a68d9f) - with back throttling disabled compute becomes unresponsive for more than 60 seconds (PG hard-coded client authentication connection timeout) - [test_sharded_ingest](https://github.com/neondatabase/neon/pull/11409/files#diff-e8d870165bd44acb9a6d8350f8640b301c1385a4108430b8d6d659b697e4a3f1) 600s -> 1200s Right now there are only 2 runners of that class, and if we decide to go with them, we have to check how much that type of runners we need, so jobs not stuck with waiting for that type of runners available. However we now decided to run those runners with governor performance instead of schedutil. This achieves almost same performance as previous runners but still achieves consistent results for same commit Related issue to activate performance governor on these runners https://github.com/neondatabase/runner/pull/138 ## Verification that it helps ### analyze runtimes on new runner for same commit Table of runtimes for the same commit on different runners in [run](https://github.com/neondatabase/neon/actions/runs/14417589789) | Run | Benchmarks (1) | Benchmarks (2) |Benchmarks (3) |Benchmarks (4) | Benchmarks (5) | |--------|--------|---------|---------|---------|---------| | 1 | 1950.37s | 6374.55s | 3646.15s | 4149.48s | 2330.22s | | 2 | - | 6369.27s | 3666.65s | 4162.42s | 2329.23s | | Delta % | - | 0,07 % | 0,5 % | 0,3 % | 0,04 % | | with governor performance | 1519.57s | 4131.62s | - | - | - | | second run gov. perf. | 1513.62s | 4134.67s | - | - | - | | Delta % | 0,3 % | 0,07 % | - | - | - | | speedup gov. performance | 22 % | 35 % | - | - | - | | current desktop class hetzner runners (main) | 1487.10s | 3699.67s | - | - | - | | slower than desktop class | 2 % | 12 % | - | - | - | In summary, the runtimes for the same commit on this hardware varies less than 1 %. --------- Co-authored-by: BodoBolero <peterbendel@neon.tech>