Bojan Serafimov
c44c8a0ea0
Add comment
2022-06-23 14:14:39 -04:00
Bojan Serafimov
9aab1d0f2b
Expand blocking scope
2022-06-22 16:04:42 -04:00
Bojan Serafimov
83dc93ab0f
Merge branch 'main' into tenant-tasks
2022-06-22 14:49:45 -04:00
KlimentSerafimov
d059e588a6
Added invariant check for project name. ( #1921 )
...
Summary: Added invariant checking for project name. Refactored ClientCredentials and TlsConfig.
* Added formatting invariant check for project name:
**\forall c \in project_name . c \in [alnum] U {'-'}.
** sni_data == <project_name>.<common_name>
* Added exhaustive tests for get_project_name.
* Refactored TlsConfig to contain common_name : Option<String>.
* Refactored ClientCredentials construction to construct project_name directly.
* Merged ProjectNameError into ClientCredsParseError.
* Tweaked proxy tests to accommodate refactored ClientCredentials construction semantics.
* [Pytests] Added project option argument to test_proxy_select_1.
* Removed project param from Api since now it's contained in creds.
* Refactored &Option<String> -> Option<&str>.
Co-authored-by: Dmitrii Ivanov <dima@neon.tech >.
2022-06-22 09:34:24 -04:00
Sergey Melnikov
6222a0012b
Migrate from CircleCI to Github Actions: python codestyle, build and tests ( #1647 )
...
Duplicate postgres and neon build and test jobs from CircleCI to Github actions.
2022-06-22 11:40:59 +03:00
bojanserafimov
1ca28e6f3c
Import basebackup into pageserver ( #1925 )
...
Allow importing basebackup taken from vanilla postgres or another pageserver via psql copy in protocol.
2022-06-21 11:04:10 -04:00
Arthur Petukhovsky
6c4d6a2183
Remove timeline_start_lsn check temporary. ( #1964 )
2022-06-21 02:02:24 +03:00
Thang Pham
37465dafe3
Add wal backpressure tests ( #1919 )
...
Resolves #1889 .
This PR adds new tests to measure the WAL backpressure's performance under different workloads.
## Changes
- add new performance tests in `test_wal_backpressure.py`
- allow safekeeper's fsync to be configurable when running tests
2022-06-20 11:40:55 -04:00
Joshua D. Drake
ec0064c442
Small README.md changes ( #1957 )
...
* Update make instructions for release and debug build. Update PostgreSQL glossary to proper version (14)
* Continued cleanup of build instructions including removal of redundancies
2022-06-20 10:05:10 -04:00
Heikki Linnakangas
83c7e6ce52
Bump vendor/postgres.
...
This brings in the change to not use a shared memory in the WAL redo
process, to avoid running out of sysv shmem segments in the page server.
Also, removal of callmemaybe bits.
2022-06-20 15:28:43 +03:00
Arthur Petukhovsky
f862373ac0
Fix WAL timeout in test_s3_wal_replay ( #1953 )
2022-06-17 20:43:54 +03:00
Arthur Petukhovsky
699f46cd84
Download WAL from S3 if it's not available in safekeeper dir ( #1932 )
...
`send_wal.rs` and `WalReader` are now async. `test_s3_wal_replay` checks that WAL can be replayed after offloaded.
2022-06-17 15:33:39 +03:00
Anastasia Lubennikova
36ee182d26
Implement page servise 'fullbackup' endpoint ( #1923 )
...
* Implement page servise 'fullbackup' endpoint that works like basebackup, but also sends relational files
* Add test_runner/batch_others/test_fullbackup.py
Co-authored-by: bojanserafimov <bojan.serafimov7@gmail.com >
2022-06-16 14:07:11 +03:00
Anastasia Lubennikova
d11c9f9fcb
Use random ports for the proxy and local pg in tests
...
Fixes #1931
Author: Dmitry Ivanov
2022-06-15 20:21:58 +03:00
Bojan Serafimov
9a9a58d52c
Fmt
2022-06-15 09:49:07 -04:00
Bojan Serafimov
865e8740a7
Add docs
2022-06-15 09:35:41 -04:00
Bojan Serafimov
1a5d1a15d0
Use tokio sleep instead
2022-06-15 09:33:56 -04:00
Bojan Serafimov
02a9883f0f
Add TenantTaskManager
2022-06-15 09:24:46 -04:00
Bojan Serafimov
ee36ca54d5
Run compaction as task
2022-06-15 09:16:57 -04:00
Bojan Serafimov
36cc6d2928
Fmt
2022-06-14 15:07:36 -04:00
Bojan Serafimov
e1a4c06918
Fix init
2022-06-14 15:05:30 -04:00
Bojan Serafimov
ec4528505e
Simplify
2022-06-14 12:38:09 -04:00
Bojan Serafimov
c79e72e835
Add runtime
2022-06-14 11:08:17 -04:00
Bojan Serafimov
a1f85715ac
WIP
2022-06-14 09:39:55 -04:00
Kirill Bulatov
d8a37452c8
Rename ZenithFeedback ( #1912 )
2022-06-11 00:44:05 +03:00
chaitanya sharma
e1336f451d
renamed .zenith data-dir to .neon.
2022-06-09 18:19:18 +02:00
Arseny Sher
a4d8261390
Save Postgres log in test_find_end_of_wal_* tests.
2022-06-09 19:16:43 +04:00
Egor Suvorov
e2a5a31595
Safekeeper HTTP router: add comment about /v1/timeline
2022-06-09 17:14:46 +02:00
Egor Suvorov
0ac0fba77a
test_runner: test Safekeeper HTTP API Auth
...
All endpoints except for POST /v1/timeline are tested, this one is not tested in any way yet.
Three attempts for each endpoint: correctly authenticated, badly authenticated, unauthenticated.
2022-06-09 17:14:46 +02:00
Egor Suvorov
a001052cdd
test_runner: SafekeeperHttpClient: support auth
2022-06-09 17:14:46 +02:00
Egor Suvorov
1f1d852204
ZenithEnvBuilder: rename pageserver_auth_enabled --> auth_enabled
2022-06-09 17:14:46 +02:00
Egor Suvorov
f7b878611a
Implement JWT authentication in Safekeeper HTTP API ( #1753 )
...
* `control_plane` crate (used by `neon_local`) now parses an `auth_enabled` bool for each Safekeeper
* If auth is enabled, a Safekeeper is passed a path to a public key via a new command line argument
* Added TODO comments to other places needing auth
2022-06-09 17:14:46 +02:00
Arseny Sher
a51b2dac9a
Don't s3 offload from newly joined safekeeper not having required WAL.
...
I made the check at launcher level with the perspective of generally moving
election (decision who offloads) there.
Also log timeline 'active' changes.
2022-06-09 18:30:16 +04:00
Thang Pham
e22d9cee3a
fix ZeroDivisionError in scripts/generate_perf_report_page ( #1906 )
...
Fixes the `ZeroDivisionError` error by adding `EPS=1e-6` when doing the calculation.
2022-06-08 09:15:12 -04:00
Arthur Petukhovsky
a01999bc4a
Replace most common remote logs with metrics ( #1909 )
2022-06-08 13:36:49 +03:00
chaitanya sharma
32e64afd54
Use better parallel build instructions in readme.md ( #1908 )
2022-06-08 11:25:37 +03:00
Kirill Bulatov
8a53472e4f
Force etcd broker keys to not to intersect
2022-06-08 11:21:05 +03:00
Dmitry Rodionov
6e26588d17
Allow to customize shutdown condition in PostgresBackend
...
Use it in PageServerHandler to check per thread shutdown condition
from thread_mgr which takes into account tenants and timelines
2022-06-07 22:11:54 +03:00
Arseny Sher
0b93253b3c
Fix leaked keepalive task in s3 offloading leader election.
...
I still don't like the surroundings and feel we'd better get away without using
election API at all, but this is a quick fix to keep CI green.
ref #1815
2022-06-07 15:17:57 +04:00
Dmitry Rodionov
7dc6beacbd
make it possible to associate thread with a tenant after thread start
2022-06-07 12:59:35 +03:00
Thang Pham
6cfebc096f
Add read/write throughput performance tests ( #1883 )
...
Part of #1467
This PR adds several performance tests that compare the [PG statistics](https://www.postgresql.org/docs/current/monitoring-stats.html ) obtained when running PG benchmarks against Neon and vanilla PG to measure the read/write throughput of the DB.
2022-06-06 12:32:10 -04:00
KlimentSerafimov
fecad1ca34
Resolving issue #1745 . Added cluster option for SNI data ( #1813 )
...
* Added project option in case SNI data is missing. Resolving issue #1745 .
* Added invariant checking for project name: if both sni_data and project_name are available then they should match.
2022-06-06 08:14:41 -04:00
bojanserafimov
92de8423af
Remove dead code ( #1886 )
2022-06-05 09:18:11 -04:00
Dmitry Rodionov
e442f5357b
unify two identical failpoints in flush_frozen_layer
...
probably is a merge artfact
2022-06-03 19:36:09 +03:00
Arseny Sher
5a723d44cd
Parametrize test_normal_work.
...
I like to run small test locally, but let's avoid duplication.
2022-06-03 20:32:53 +04:00
Kirill Bulatov
2623193876
Remove pageserver_connstr from WAL stream logic
2022-06-03 17:30:36 +03:00
Arseny Sher
70a53c4b03
Get backup test_safekeeper_normal_work, but skip by default.
...
It is handy for development.
2022-06-03 16:12:14 +04:00
Arseny Sher
9e108102b3
Silence etcd safekeeper info key parse errors.
...
When we subscribe to everything, it is ok to receive not only safekeeper
timeline updates.
2022-06-03 16:12:14 +04:00
huming
9c846a93e8
chore(doc)
2022-06-03 14:24:27 +03:00
Kirill Bulatov
c5007d3916
Remove unused module
2022-06-03 00:23:13 +03:00
