9ff122835f
This clarifies - I hope - the abstractions between Repository and ObjectRepository. The ObjectTag struct was a mix of objects that could be accessed directly through the public Timeline interface, and also objects that were created and used internally by the ObjectRepository implementation and not supposed to be accessed directly by the callers. With the RelishTag separaate from ObjectTag, the distinction is more clear: RelishTag is used in the public interface, and ObjectTag is used internally between object_repository.rs and object_store.rs, and it contains the internal metadata object types. One awkward thing with the ObjectTag struct was that the Repository implementation had to distinguish between ObjectTags for relations, and track the size of the relation, while others were used to store "blobs". With the RelishTags, some relishes are considered "non-blocky", and the Repository implementation is expected to track their sizes, while others are stored as blobs. I'm not 100% happy with how RelishTag captures that either: it just knows that some relish kinds are blocky and some non-blocky, and there's an is_block() function to check that. But this does enable size-tracking for SLRUs, allowing us to treat them more like relations. This changes the way SLRUs are stored in the repository. Each SLRU segment, e.g. "pg_clog/0000", "pg_clog/0001", are now handled as a separate relish. This removes the need for the SLRU-specific put_slru_truncate() function in the Timeline trait. SLRU truncation is now handled by caling put_unlink() on the segment. This is more in line with how PostgreSQL stores SLRUs and handles their trunction. The SLRUs are "blocky", so they are accessed one 8k page at a time, and repository tracks their size. I considered an alternative design where we would treat each SLRU segment as non-blocky, and just store the whole file as one blob. Each SLRU segment is up to 256 kB in size, which isn't that large, so that might've worked fine, too. One reason I didn't do that is that it seems better to have the WAL redo routines be as close as possible to the PostgreSQL routines. It doesn't matter much in the repository, though; we have to track the size for relations anyway, so there's not much difference in whether we also do it for SLRUs. While working on this, I noticed that the CLOG and MultiXact redo code did not handle wraparound correctly. We need to fix that, but for now, I just commented them out with a FIXME comment.
Zenith test runner
This directory contains integration tests.
Prerequisites:
- Python 3.6 or later
- Dependencies: install them via
pipenv install. Note that Debian/Ubuntu packages are stale, as it commonly happens, so manual installation is not recommended. Runpipenv shellto activate the venv. - Zenith and Postgres binaries
- See the root README.md for build directions
- Tests can be run from the git tree; or see the environment variables below to run from other directories.
- The zenith git repo, including the postgres submodule (for some tests, e.g. pg_regress)
Test Organization
The tests are divided into a few batches, such that each batch takes roughly the same amount of time. The batches can be run in parallel, to minimize total runtime. Currently, there are only two batches:
- test_batch_pg_regress: Runs PostgreSQL regression tests
- test_others: All other tests
Running the tests
Because pytest will search all subdirectories for tests, it's easiest to
run the tests from within the test_runner directory.
Test state (postgres data, pageserver state, and log files) will
be stored under a directory test_output.
You can run all the tests with:
pytest
If you want to run all the tests in a particular file:
pytest test_pgbench.py
If you want to run all tests that have the string "bench" in their names:
pytest -k bench
Useful environment variables:
ZENITH_BIN: The directory where zenith binaries can be found.
POSTGRES_DISTRIB_DIR: The directory where postgres distribution can be found.
TEST_OUTPUT: Set the directory where test state and test output files
should go.
TEST_SHARED_FIXTURES: Try to re-use a single pageserver for all the tests.
Let stdout and stderr go to the terminal instead of capturing them:
pytest -s ...
(Note many tests capture subprocess outputs separately, so this may not
show much.)
Exit after the first test failure:
pytest -x ...
(there are many more pytest options; run pytest -h to see them.)
Building new tests
The tests make heavy use of pytest fixtures. You can read about how they work here: https://docs.pytest.org/en/stable/fixture.html
Essentially, this means that each time you see a fixture named as an input parameter, the function with that name will be run and passed as a parameter to the function.
So this code:
def test_something(zenith_cli, pg_bin):
pass
... will run the fixtures called zenith_cli and pg_bin and deliver those results to the test function.
Fixtures can't be imported using the normal python syntax. Instead, use this:
pytest_plugins = ("fixtures.something")
That will make all the fixtures in the fixtures/something.py file available.
Anything that's likely to be used in multiple tests should be built into a fixture.
Note that fixtures can clean up after themselves if they use the yield syntax.
Cleanup will happen even if the test fails (raises an unhandled exception).
Python destructors, e.g. __del__() aren't recommended for cleanup.
Code quality
Before submitting a patch, please consider:
- Writing a couple of docstrings to clarify the reasoning behind a new test.
- Running
flake8(or a linter of your choice, e.g.pycodestyle) and fixing possible defects, if any. - Formatting the code with
yapf -r -i .(TODO: implement an opt-in pre-commit hook for that). - (Optional) Typechecking the code with
mypy .. Currently this mostly affectsfixtures/zenith_fixtures.py.
The tools can be installed with pipenv install --dev.