This provides a pytest fixture to record metrics from pytest tests. The
The recorded metrics are printed out at the end of the tests.
As a starter, this includes on small test, using pgbench. It prints out
three metrics: the initialization time, runtime of 5000 xacts, and the
repository size after the tests.
For a CI build, storing incremental build data just makes the cache
bigger, for minimal gain.
Also, for Rust < 1.52.1 there are incremental compilation bugs. CircleCI
is currently building on 1.51.
This only affects the debug build; incremental compilation isn't used on
the release build.
Add parameters to specify which kind of build; run a debug and release
variant for each job.
Eventually this will be too many jobs, but for now this is a nice start.
Also, bump the cache string to "v02" so we don't mix up our cache output
with other branches.
- The 'pageserver' fixture now sets up the repository and starts up
the Page Server automatically. In other words, the 'pageserver'
fixture provides a Page Server that's up and running and ready to
use in tests.
- The 'pageserver' fixture now also creates a branch called 'empty',
right after initializing the repository. By convention, all the
tests start by createing a new branch off 'empty' for the test. This
allows running all the tests against the same Page Server
concurrently. (I haven't tested that though. pytest doensn't
provide an option to run tests in parallel but there are extensions
for that.)
- Remove the 'zen_simple' fixture. Now that 'pageserver' provides
server that's up and running, it's pretty simple to use the
'pageserver' and 'postgres' fixtures directly.
- Don't assume host name or ports in the tests. They now use the
fields in the fixtures for that. That allows assigning the ports
dynamically, making it possible to run multiple page servers in
parallel, or running the tests in parallel with another page
server. This commit still hard codes the Page Server's port in the
fixture, though, so more work is needed to actually make it
possible.
- I made some changes to the 'postgres' fixture in commit 532918e13d,
which broke the other tests. Fix them.
- Divide the tests into two "batches" of roughly equal runtime, which
can be run in parallel
- Merge the 'test_file' and 'test_filter' options in CircleCI config
into one 'test_selection' option, for simplicity.
This patch started as an effort to support CLI working against remote
pageserver, but turned into a pretty big refactoring.
* CLI now does not look into repository files directly. New commands
'branch_create' and 'identify_system' were introduced into page_service to
support that.
* Branch management that was scattered between local_env and
zenith/main.rs is moved into pageserver/branches.rs. That code could better fit
in Repository/Timeline impl, but I'll leave that for a different patch.
* All tests-related code from local_env went into integration_tests/src/lib.rs as an
extension to PostgresNode trait.
* Paths-generating functions were concentrated around corresponding config
types (LocalEnv and PageserverConf).
Mostly we're not testing python code, so verbose python tracebacks are
unhelpful. Add --tb=short to the pytest args to cut down on the noise.
To override this during testing, set the "extra_params" parameter on the
circleci job to "--tb=auto" or "--tb=long".
The rust cache is growing dramatically. Change the cache key to start
over.
The weird "v98" was something I'd intended to reset before landing the
circleci config. Do the sane thing and start over at v01. The intent is
that we just increment the number each time something gets broken.
default(medium): 2 CPUs, 4GB RAM.
xlarge: 8 CPUs, 16GB RAM.
Some build jobs are getting killed with signal 9. I'm guessing that this
is probably an OOM condition...
Fetching the postgres submodule is one of the more expensive steps of
the build. Doing a shallow clone ("--depth 1") should save some time and
a lot of network bandwidth.
This does the postgres & rust builds, caching the results, and preserves
its outputs in a "workspace" for downstream test jobs (which can run in
parallel).
Pytest jobs are parameterized, so adding new pytest-based tests requires
only adding a new job to the "workflows" section at the end.
This could use some optimization:
- The "apt-get install" step is quite slow.
- The rust build step will always happen, even if only unrelated changes
are present (e.g. modified a python test file)
- Saving/restoring the rust cache (/target) is very slow (it contains
1.3GB of data)
- Saving the workspace is very slow.
- The "install" step is ugly; postgres and rust artifacts could take a
much better form.
