39d86ed29e
debug branch failure
2022-07-09 00:42:45 +03:00
60e5dc10e6
postgres_ffi/wal_generate: use 'craft' instead of 'generate'
...
It does very fine-tuned byte-to-byte WAL crafting, not a sloppy generation.
Hence 'craft' sounds like a better description.
2022-07-08 18:30:56 +02:00
80b7a3b51a
Test what happens when XLOG_SWITCH ends on page boundary, fix #1991
2022-07-08 15:37:26 +02:00
85bda437de
postgres_ffi/wal_generate: add last_wal_record_xlog_switch and use it in tests
...
Fix #1190 : WalDecoder did not return correct LSN of the next record after
processing a XLOG_SWITCH record
2022-07-08 15:37:26 +02:00
bcdee3d3b5
test_runner: add test_crafted_wal_end.py
...
For some reason both non-`simple` tests spend about 10 seconds
in the post-restart `INSERT INTO` query on my machine, see #2023
2022-07-08 13:56:37 +02:00
00c26ff3a3
Bring periodic perf tests on GitHub back ( #2037 )
...
* test/fixtures: fix DeprecationWarning
* workflows/benchmarking: increase timeout
* test: switch pgbench to default(simple) query mode
* test/performance: ensure we don't have tables that we're creating
* workflows/pg_clients: remove unused env var
* workflows/benchmarking: change platform name
2022-07-07 19:53:23 +01:00
ec0faf3ac6
retry timeline delete
2022-07-07 21:20:04 +03:00
1a5af6d7a5
extend detach/delete tests
2022-07-07 21:20:04 +03:00
168214e0b6
use tenant status endpoint to check whether timelines were downloaded or not
2022-07-07 21:20:04 +03:00
e1e24336b7
review adjustments, bring back timeline_detach and rename it to timeline_delete
2022-07-07 21:20:04 +03:00
4c54e4b37d
switch to per-tenant attach/detach
...
download operations of all timelines for one tenant are now grouped
together so when attach is invoked pageserver downloads all of them
and registers them in a single apply_sync_status_update call so
branches can be used safely with attach/detach
2022-07-07 21:20:04 +03:00
4a96259bdd
Add export/import test ( #2036 )
2022-07-06 13:45:26 -04:00
8fabdc6708
Add tests with concurrent computes.
...
Removes test_restart_compute, as added test_compute_restarts is stronger.
2022-07-06 18:07:29 +04:00
32560e75d2
Enable relocation test ( #1974 )
2022-07-05 08:27:57 -04:00
05f6a1394d
Add tests for different Postgres client libraries ( #2008 )
...
* Add tests for different postgres clients
* test/fixtures: sanitize test name for test_output_dir
* test/fixtures: do not look for etcd before runtime
* Add workflow for testing Postgres client libraries
2022-07-05 12:22:58 +01:00
d29c545b5d
Gc/compaction thread pool, take 2 ( #1933 )
...
Decrease the number of pageserver threads by running gc and compaction in a blocking tokio thread pool
2022-07-05 02:06:40 -04:00
65704708fa
remove unused imports, make more use of pathlib.Path
2022-07-01 18:56:51 +03:00
f09c09438a
Fix gc after import
2022-07-01 11:10:49 +03:00
8a714f1ebf
Add coverage to GH actions and rework part of them ( #1987 )
2022-06-27 19:15:56 +03:00
84b9fcbbd5
Increase a few test timeouts ( #1977 )
2022-06-23 11:51:56 -04:00
3c2b03cd87
Update timeline size on dropdb. Add the test ( #1973 )
...
In addition, fix database size calculation:
count not only main fork of the relation, but also vm and fsm.
2022-06-23 12:28:12 +03:00
d059e588a6
Added invariant check for project name. ( #1921 )
...
Summary: Added invariant checking for project name. Refactored ClientCredentials and TlsConfig.
* Added formatting invariant check for project name:
**\forall c \in project_name . c \in [alnum] U {'-'}.
** sni_data == <project_name>.<common_name>
* Added exhaustive tests for get_project_name.
* Refactored TlsConfig to contain common_name : Option<String>.
* Refactored ClientCredentials construction to construct project_name directly.
* Merged ProjectNameError into ClientCredsParseError.
* Tweaked proxy tests to accommodate refactored ClientCredentials construction semantics.
* [Pytests] Added project option argument to test_proxy_select_1.
* Removed project param from Api since now it's contained in creds.
* Refactored &Option<String> -> Option<&str>.
Co-authored-by: Dmitrii Ivanov <dima@neon.tech >.
2022-06-22 09:34:24 -04:00
1ca28e6f3c
Import basebackup into pageserver ( #1925 )
...
Allow importing basebackup taken from vanilla postgres or another pageserver via psql copy in protocol.
2022-06-21 11:04:10 -04:00
37465dafe3
Add wal backpressure tests ( #1919 )
...
Resolves #1889 .
This PR adds new tests to measure the WAL backpressure's performance under different workloads.
## Changes
- add new performance tests in `test_wal_backpressure.py`
- allow safekeeper's fsync to be configurable when running tests
2022-06-20 11:40:55 -04:00
f862373ac0
Fix WAL timeout in test_s3_wal_replay ( #1953 )
2022-06-17 20:43:54 +03:00
699f46cd84
Download WAL from S3 if it's not available in safekeeper dir ( #1932 )
...
`send_wal.rs` and `WalReader` are now async. `test_s3_wal_replay` checks that WAL can be replayed after offloaded.
2022-06-17 15:33:39 +03:00
36ee182d26
Implement page servise 'fullbackup' endpoint ( #1923 )
...
* Implement page servise 'fullbackup' endpoint that works like basebackup, but also sends relational files
* Add test_runner/batch_others/test_fullbackup.py
Co-authored-by: bojanserafimov <bojan.serafimov7@gmail.com >
2022-06-16 14:07:11 +03:00
d11c9f9fcb
Use random ports for the proxy and local pg in tests
...
Fixes #1931
Author: Dmitry Ivanov
2022-06-15 20:21:58 +03:00
e1336f451d
renamed .zenith data-dir to .neon.
2022-06-09 18:19:18 +02:00
0ac0fba77a
test_runner: test Safekeeper HTTP API Auth
...
All endpoints except for POST /v1/timeline are tested, this one is not tested in any way yet.
Three attempts for each endpoint: correctly authenticated, badly authenticated, unauthenticated.
2022-06-09 17:14:46 +02:00
a001052cdd
test_runner: SafekeeperHttpClient: support auth
2022-06-09 17:14:46 +02:00
1f1d852204
ZenithEnvBuilder: rename pageserver_auth_enabled --> auth_enabled
2022-06-09 17:14:46 +02:00
a51b2dac9a
Don't s3 offload from newly joined safekeeper not having required WAL.
...
I made the check at launcher level with the perspective of generally moving
election (decision who offloads) there.
Also log timeline 'active' changes.
2022-06-09 18:30:16 +04:00
6cfebc096f
Add read/write throughput performance tests ( #1883 )
...
Part of #1467
This PR adds several performance tests that compare the [PG statistics](https://www.postgresql.org/docs/current/monitoring-stats.html ) obtained when running PG benchmarks against Neon and vanilla PG to measure the read/write throughput of the DB.
2022-06-06 12:32:10 -04:00
e442f5357b
unify two identical failpoints in flush_frozen_layer
...
probably is a merge artfact
2022-06-03 19:36:09 +03:00
5a723d44cd
Parametrize test_normal_work.
...
I like to run small test locally, but let's avoid duplication.
2022-06-03 20:32:53 +04:00
70a53c4b03
Get backup test_safekeeper_normal_work, but skip by default.
...
It is handy for development.
2022-06-03 16:12:14 +04:00
a91e0c299d
Reproduce etcd parsing bug in Python tests
2022-06-03 00:23:13 +03:00
90e2c9ee1f
Rename zenith to neon in python tests ( #1871 )
2022-06-02 16:21:28 -04:00
e5cb727572
Replace callmemaybe with etcd subscriptions on safekeeper timeline info
2022-06-01 16:07:04 +03:00
b1b67cc5a0
improve test normal work to start several computes
2022-05-31 22:42:11 +03:00
36281e3b47
Extend test_wal_backup with compute restart.
2022-05-30 13:57:17 +04:00
e014cb6026
rename zenith.zenith_tenant to neon.tenant_id in test
2022-05-30 12:24:44 +03:00
67d6ff4100
Rename custom GUCs:
...
- zenith.zenith_tenant -> neon.tenant_id
- zenith.zenith_timeline -> neon.timeline_id
2022-05-30 11:11:01 +03:00
6a867bce6d
Rename 'zenith_admin' role to 'cloud_admin'
2022-05-30 11:11:01 +03:00
751f1191b4
Rename 'wal_acceptors' GUC to 'safekeepers'
2022-05-30 11:11:01 +03:00
3accde613d
Rename contrib/zenith to contrib/neon. Rename custom GUCs:
...
- zenith.page_server_connstring -> neon.pageserver_connstring
- zenith.zenith_tenant -> neon.tenantid
- zenith.zenith_timeline -> neon.timelineid
- zenith.max_cluster_size -> neon.max_cluster_size
2022-05-30 11:11:01 +03:00
4b4d3073b8
Fix misc typos
2022-05-28 14:56:23 +03:00
f1c51a1267
Fix typos
2022-05-28 14:02:05 +03:00
757746b571
Fix test_pageserver_http_get_wal_receiver_success flaky test. ( #1786 )
...
Fixes #1768 .
## Context
Previously, to test `get_wal_receiver` API, we make run some DB transactions then call the API to check the latest message's LSN from the WAL receiver. However, this test won't work because it's not guaranteed that the WAL receiver will get the latest WAL from the postgres/safekeeper at the time of making the API call.
This PR resolves the above issue by adding a "poll and wait" code that waits to retrieve the latest data from the WAL receiver.
This PR also fixes a bug that tries to compare two hex LSNs, should convert to number before the comparison. See: https://github.com/neondatabase/neon/issues/1768#issuecomment-1133752122 .
2022-05-27 13:33:53 -04:00
Dmitry Rodionov
Egor Suvorov
Alexander Bayandin
bojanserafimov
Arthur Petukhovsky
Kirill Bulatov
Anastasia Lubennikova
KlimentSerafimov
Thang Pham
chaitanya sharma
Arseny Sher
Kirill Bulatov
Anastasia Lubennikova
Heikki Linnakangas
Kian-Meng Ang