Switch our python package management solution to poetry.

Mainly because it has better support for installing the packages from
different python versions.

It also has better dependency resolver than Pipenv. And supports modern
standard for python dependency management. This includes usage of
pyproject.toml for project specific configuration instead of per
tool conf files. See following links for details:
 https://pip.pypa.io/en/stable/reference/build-system/pyproject-toml/
 https://www.python.org/dev/peps/pep-0518/

test_runner/README.md

@@ -22,23 +22,24 @@ runtime. Currently, there are only two batches:
 
 ### Running the tests
 
-Because pytest will search all subdirectories for tests, it's easiest to
-run the tests from within the `test_runner` directory.
+There is a wrapper script to invoke pytest: `./scripts/pytest`.
+It accepts all the arguments that are accepted by pytest.
+Depending on your installation options pytest might be invoked directly.
 
 Test state (postgres data, pageserver state, and log files) will
 be stored under a directory `test_output`.
 
 You can run all the tests with:
 
-`pipenv run pytest`
+`./scripts/pytest`
 
 If you want to run all the tests in a particular file:
 
-`pipenv run pytest test_pgbench.py`
+`./scripts/pytest test_pgbench.py`
 
 If you want to run all tests that have the string "bench" in their names:
 
-`pipenv run pytest -k bench`
+`./scripts/pytest -k bench`
 
 Useful environment variables:
 
@@ -53,12 +54,12 @@ should go.
 `RUST_LOG`: logging configuration to pass into Zenith CLI
 
 Let stdout, stderr and `INFO` log messages go to the terminal instead of capturing them:
-`pytest -s --log-cli-level=INFO ...`
+`./scripts/pytest -s --log-cli-level=INFO ...`
 (Note many tests capture subprocess outputs separately, so this may not
 show much.)
 
 Exit after the first test failure:
-`pytest -x ...`
+`./scripts/pytest -x ...`
 (there are many more pytest options; run `pytest -h` to see them.)
 
 ### Writing a test

test_runner/batch_others/test_remote_storage.py

@@ -1,5 +1,5 @@
 # It's possible to run any regular test with the local fs remote storage via
-# env ZENITH_PAGESERVER_OVERRIDES="remote_storage={local_path='/tmp/zenith_zzz/'}" pipenv ......
+# env ZENITH_PAGESERVER_OVERRIDES="remote_storage={local_path='/tmp/zenith_zzz/'}" poetry ......
 
 import time, shutil, os
 from contextlib import closing

test_runner/fixtures/log_helper.py

@@ -3,11 +3,11 @@ import logging.config
 """
 This file configures logging to use in python tests.
 Logs are automatically captured and shown in their
-own section after all tests are executed. 
+own section after all tests are executed.
 
 To see logs for all (even successful) tests, run
 pytest with the following command:
-- `pipenv run pytest -n8 -rA`
+- `poetry run pytest -n8 -rA`
 
 Other log config can be set in pytest.ini file.
 You can add `log_cli = true` to it to watch
@@ -34,7 +34,7 @@ LOGGING = {
 
 def getLogger(name='root') -> logging.Logger:
     """Method to get logger for tests.
-    
+
     Should be used to get correctly initialized logger. """
     return logging.getLogger(name)
 

test_runner/fixtures/zenith_fixtures.py

@@ -334,7 +334,7 @@ class AuthKeys:
 class MockS3Server:
     """
     Starts a mock S3 server for testing on a port given, errors if the server fails to start or exits prematurely.
-    Relies that `pipenv` and `moto` server are installed, since it's the way the tests are run.
+    Relies that `poetry` and `moto` server are installed, since it's the way the tests are run.
 
     Also provides a set of methods to derive the connection properties from and the method to kill the underlying server.
     """
@@ -344,7 +344,7 @@ class MockS3Server:
     ):
         self.port = port
 
-        self.subprocess = subprocess.Popen([f'pipenv run moto_server s3 -p{port}'], shell=True)
+        self.subprocess = subprocess.Popen([f'poetry run moto_server s3 -p{port}'], shell=True)
         error = None
         try:
             return_code = self.subprocess.poll()