From 954859f6c5648aa351b5c4a0b05b3db0f369a0ab Mon Sep 17 00:00:00 2001 From: Dmitry Rodionov Date: Thu, 5 May 2022 13:15:53 +0300 Subject: [PATCH] add readme for performance tests with the current state of things --- test_runner/performance/README.md | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100644 test_runner/performance/README.md diff --git a/test_runner/performance/README.md b/test_runner/performance/README.md new file mode 100644 index 0000000000..7812c73f0c --- /dev/null +++ b/test_runner/performance/README.md @@ -0,0 +1,23 @@ +# What performance tests do we have and how we run them + +Performanse tests are build using infrastructure of our usual python integration tests. + +## Tests that are run against local installation + +Most off the performance tests run against local installation. This causes some problems because safekeeper(s) and a pageserver share resources of one single host and one underlyinng disk. + +These tests are run in CI in the same environment as the usual integration tests. So environment may not yield comarable results because this is the machine that CI provider gives us. + +## Remote tests + +There are a few tests that marked with `pytest.mark.remote_cluster`. These tests do not use local installation and onnly need a connection string to run. So they can be used for every postgresql comatible database. Currenntly these tests are run against our staging daily. Staging is not an isolated environment, so it adds to possible noise due to activity of other clusters. + +## Noise + +All tests run only once. Usually to obtain more consistent performance numbers test is performed multiple times and then some statistics is applied to results, like min/max/avg/median etc. + +## Results collection + +Local tests results for main branch and results of daily performance tests are stored in neon cluster deployed in production environment and there is a grafana dashboard that visualizes the results. Here is the [dashboard](https://observer.zenith.tech/d/DGKBm9Jnz/perf-test-results?orgId=1). The main problem with it is the unavailability to point at particular commits though the data for that is available in the database. Needs some tweaking from someone who knows Grafana tricks. + +There is also an inconsistency in test naming. Test name should be the same across platforms and results can be differentiated by the platform field. But now platform is sometimes included in test name because of the way how parametrization works in pytest. Ie there is a platform switch in the dashboard with zenith-local-ci and zenith-staging variants. I e some tests under zenith-local-ci value for a platform switch are displayed as `Test test_runner/performance/test_bulk_insert.py::test_bulk_insert[vanilla]` and `Test test_runner/performance/test_bulk_insert.py::test_bulk_insert[zenith]` which is highly confusing.