Commit Graph

998 Commits

Author SHA1 Message Date
Konstantin Knizhnik
2e54dcdffe Fix deadlock in buffered repo 2021-12-28 19:20:28 +03:00
Konstantin Knizhnik
1ca9bea7c0 Reduce toast segment size to 2000 2021-12-15 18:19:45 +03:00
Konstantin Knizhnik
945e83a6ba Add nonblock.rs 2021-12-15 00:05:26 +03:00
Konstantin Knizhnik
eb2323aedd Use walredo from master 2021-12-08 12:27:27 +03:00
Konstantin Knizhnik
cb6e231ea9 Separate checkpointing and page reconstruction 2021-12-06 12:24:13 +03:00
Konstantin Knizhnik
2460f44e30 Fix Clippy warnings' 2021-12-02 14:21:36 +03:00
Konstantin Knizhnik
863c04b0a6 Use latest YAKV version 2021-12-02 13:01:27 +03:00
Konstantin Knizhnik
e4b89a1849 Use parking_lot synchronization primitives 2021-11-30 18:28:35 +03:00
Konstantin Knizhnik
5ad82418a9 Add upload thread 2021-11-24 18:04:48 +03:00
Konstantin Knizhnik
92562145c0 Change toast store API 2021-11-23 11:34:32 +03:00
Konstantin Knizhnik
915001c67e Fix clippy warnings 2021-11-22 20:05:34 +03:00
Konstantin Knizhnik
13f9565ff8 Fix indentation 2021-11-22 19:22:59 +03:00
Konstantin Knizhnik
f73d043a8b Use COW version of YAKV 2021-11-22 19:22:39 +03:00
Konstantin Knizhnik
8fda7a6183 Fix indentation 2021-11-17 12:50:46 +03:00
Konstantin Knizhnik
4acd292717 Use BRIN to optimize GC 2021-11-17 12:50:25 +03:00
Konstantin Knizhnik
b365a075f4 Save materialized pages 2021-11-17 12:16:14 +03:00
Konstantin Knizhnik
6311135d73 Save materialized pages 2021-11-17 12:16:02 +03:00
Konstantin Knizhnik
ee29446edc Add BRIN index for checkpointer 2021-11-12 17:20:17 +03:00
Konstantin Knizhnik
d2e5e0e728 Fix compression 2021-11-11 00:16:37 +03:00
Konstantin Knizhnik
3b471494ff Add import/export functions from buffered stotage to files with layeres 2021-11-10 09:48:28 +03:00
Konstantin Knizhnik
9947de4a2a Fix issues with garbage collector 2021-11-03 12:15:24 +03:00
Konstantin Knizhnik
a3e94e888a Implement garbage collector for buffered repository 2021-10-30 13:10:04 +03:00
Konstantin Knizhnik
e6f33a5cd0 Rewrite TOAST to use the same tree as main index 2021-10-29 17:00:09 +03:00
Konstantin Knizhnik
2dd35b1fbe Fix indentation 2021-10-27 19:37:50 +03:00
Konstantin Knizhnik
ce779cc754 Use delayed commit in buffered_repo 2021-10-27 19:37:23 +03:00
Konstantin Knizhnik
497258c6fe Do not produce error in get_page_at_lsn on missed page 2021-10-26 20:07:11 +03:00
Konstantin Knizhnik
0b6008012d Apply cargo fmt 2021-10-22 19:52:05 +03:00
Konstantin Knizhnik
d35fc20181 Implement buffered repository 2021-10-22 19:50:59 +03:00
Egor Suvorov
e42c884c2b test_runner/README: add note on capturing logs (#778)
Became actual after #674
2021-10-20 01:55:49 +03:00
Egor Suvorov
eb706bc9f4 Force yapf (Python code formatter) in CI (#772)
* Add yapf run to CircleCI
* Pin yapf version
* Enable `SPLIT_ALL_TOP_LEVEL_COMMA_SEPARATED_VALUES` setting
* Reformat all existing code with slight manual adjustments
* test_runner/README: note that yapf is forced
2021-10-19 20:13:47 +03:00
Dmitry Rodionov
798df756de suppress FileNotFound exception instead of missing_ok=True because the latter is added in python 3.8 and we claim to support >3.6 2021-10-19 17:13:42 +03:00
Dmitry Rodionov
732d13fe06 use cached-property package because python<3.8 doesnt have cached_property in functools 2021-10-19 17:13:42 +03:00
Heikki Linnakangas
feae7f39c1 Support read-only nodes
Change 'zenith.signal' file to a human-readable format, similar to
backup_label. It can contain a "PREV LSN: %X/%X" line, or a special
value to indicate that it's OK to start with invalid LSN ('none'), or
that it's a read-only node and generating WAL is forbidden
('invalid').

The 'zenith pg create' and 'zenith pg start' commands now take a node
name parameter, separate from the branch name. If the node name is not
given, it defaults to the branch name, so this doesn't break existing
scripts.

If you pass "foo@<lsn>" as the branch name, a read-only node anchored
at that LSN is created. The anchoring is performed by setting the
'recovery_target_lsn' option in the postgresql.conf file, and putting
the server into standby mode with 'standby.signal'.

We no longer store the synthetic checkpoint record in the WAL segment.
The postgres startup code has been changed to use the copy of the
checkpoint record in the pg_control file, when starting in zenith
mode.
2021-10-19 09:48:12 +03:00
Heikki Linnakangas
c2b468c958 Separate node name from the branch name in ComputeControlPlane
This is in preparation for supporting read-only nodes. You can launch
multiple read-only nodes on the same brach, so we need an identifier
for each node, separate from the branch name.
2021-10-19 09:48:10 +03:00
Heikki Linnakangas
e272a380b4 On new repo, start writing WAL only after the initial checkpoint record.
Previously, the first WAL record on the 'main' branch overwrote the
initial checkpoint record, with invalid 'xl_prev'. That's harmless, but
also pretty ugly. I bumped into this while I was trying to tighen up the
checks for when a valid 'prev_lsn' is required. With this patch, the
first WAL record gets a valid 'xl_prev' value. It doesn't matter much
currently, but let's be tidy.
2021-10-19 09:48:04 +03:00
anastasia
0dc7a3fc15 Change tenant_mgr to use TenantState.
It allows to avoid locking entire TENANTS list while one tenant is bootstrapping
and prepares the code for remote storage integration.
2021-10-18 15:40:06 +03:00
Egor Suvorov
a1bc0ada59 Dockerfile: remove wal_acceptor alias for safekeeper (#743) 2021-10-18 14:56:30 +03:00
Kirill Bulatov
e9b5224a8a Fix toml serde gotchas 2021-10-18 14:14:27 +03:00
Heikki Linnakangas
bdd039a9ee S3 DELETE call returns 204, not 200.
According to the S3 API docs, the DELETE call returns code "204 No content"
on success.
2021-10-17 16:21:58 +03:00
Heikki Linnakangas
b405eef324 Avoid writing the metadata file when it hasn't changed. 2021-10-17 14:54:39 +03:00
Kirill Bulatov
ba557d126b React on sigint 2021-10-15 21:24:24 +03:00
Patrick Insinger
2dde20a227 Bump MSRV to 1.55 2021-10-15 09:10:08 -07:00
Kirill Bulatov
4ade0bb41c Refactor upload/download_relish function signatures.
This makes them more generic, by taking any Read / Write trait
implementation, instead of operating directly on a a file.
2021-10-15 11:34:15 +03:00
Stas Kelvich
100da024b6 expose pageserver http socket in docker 2021-10-15 00:26:38 +03:00
Arseny Sher
de744a44dd Add /timeline http request to safekeeper returning its status.
Which is mainly generational state (terms) and useful LSNs.

Also add /status basic healthcheck request which is now used in tests to
determine the safekeeper is up; this fixes #726.

ref #115
2021-10-14 19:02:38 +03:00
Heikki Linnakangas
0e026371ec Optimize WAL decoding slightly.
This adds a fast-path for the common case that the record doesn't
cross a page boundary. We now split off a new Bytes directly from the
original input buffer in that case, instead of copying the record to a
new BytesMut. Shaves about 5% of the page server's CPU time on my
laptop, in the 'test_bulk_insert' test.
2021-10-14 14:21:23 +03:00
Arthur Petukhovsky
4b87acb1f6 Use logging in python tests (#674)
* Use logging in python tests

* Use f-strings for logs

* Don't log test output while running

* Use only pytest logging handler

* Add more info about pytest logging
2021-10-14 13:10:09 +03:00
Dmitry Ivanov
43957f4401 [cross-repo-ci] Use solely commit hash to test PRs in CI
See #744 for the discussion.
2021-10-13 17:16:02 +03:00
Heikki Linnakangas
8a4f092e82 Skip syncing the temp initdb installation.
Doesn't make much difference on my laptop with SSD, but every little
helps, and with a slower disk it might be noticeable.
2021-10-13 16:59:00 +03:00
Egor Suvorov
6b6b3f68be Safekeeper metrics refactor (#747) 2021-10-13 16:28:24 +03:00