Commit Graph

597 Commits

Author SHA1 Message Date
Stas Kelvich
2b33894e7b few more review fixes 2021-07-19 14:52:41 +03:00
Stas Kelvich
a118557331 review fixes 2021-07-19 14:52:41 +03:00
Stas Kelvich
8ec234ba78 fix tokio features set for proxy standalone build 2021-07-19 14:52:41 +03:00
Stas Kelvich
70926adaba allow to specify redirect_uri in proxy 2021-07-19 14:52:41 +03:00
Stas Kelvich
560f088f05 fix query_string parsing for mgmt callback 2021-07-19 14:52:41 +03:00
Stas Kelvich
aa404b60fe change mgmt json format; add cli flags 2021-07-19 14:52:41 +03:00
Stas Kelvich
1b6d99db7c unfreeze client session upon callback 2021-07-19 14:52:41 +03:00
Stas Kelvich
605b90c6c7 do an actual proxy pass 2021-07-19 14:52:41 +03:00
Stas Kelvich
6f747893be quick mgmp api 2021-07-19 14:52:41 +03:00
Stas Kelvich
dab34c3dd6 distinguish between new and old users 2021-07-19 14:52:41 +03:00
Stas Kelvich
bf45bef284 md5 auth for postgres_backend.rs 2021-07-19 14:52:41 +03:00
Konstantin Knizhnik
d55095ab21 [refer #331] Move initialization of checkpoint object into import_timeline_from_postgres_datadir 2021-07-16 18:43:07 +03:00
Konstantin Knizhnik
a048e0c7c1 Revert assert cechk for advancing prev_record_lsn because in case of creation of branch on some older LSN we can not provide proper xl_prev value 2021-07-16 18:43:07 +03:00
Konstantin Knizhnik
9436c4161f Adding comment to prev_record_lsn update 2021-07-16 18:43:07 +03:00
Konstantin Knizhnik
e74b06d999 Pass prev_record_ptr through zenith.signal file to compute node 2021-07-16 18:43:07 +03:00
Konstantin Knizhnik
f6705b7a7d Fix TimestampTz type to i64 to be compatbile with Postgres 2021-07-16 18:43:07 +03:00
Konstantin Knizhnik
386403dcd1 Hold CLOG for prepared transactions 2021-07-16 18:43:07 +03:00
Heikki Linnakangas
46e613f423 Fix typos 2021-07-16 18:43:07 +03:00
Konstantin Knizhnik
56da87cadc Merge with main branch 2021-07-16 18:43:07 +03:00
Konstantin Knizhnik
dcabe694ee Fix usage of put_page_image in save_xlog_dbase_create 2021-07-16 18:43:07 +03:00
Konstantin Knizhnik
842419b91f Do not update relation metadata in get_page_at_lsn 2021-07-16 18:43:07 +03:00
Konstantin Knizhnik
3cded20662 Refactring after Heikki review 2021-07-16 18:43:07 +03:00
Konstantin Knizhnik
eb0a56eb22 Replay non-relational WAL records on page server 2021-07-16 18:43:07 +03:00
Dmitry Rodionov
8a541147e2 run cargo generate-lockfile
It removes remaining issues with running cargo audit. There was one
error and one warning:

Crate:         tokio
Version:       1.5.0
Title:         Task dropped in wrong thread when aborting `LocalSet` task
Date:          2021-07-07
ID:            RUSTSEC-2021-0072
URL:           https://rustsec.org/advisories/RUSTSEC-2021-0072
Solution:      Upgrade to >=1.5.1, <1.6.0 OR >=1.6.3, <1.7.0 OR >=1.7.2, <1.8.0 OR >=1.8.1

Crate:         cpuid-bool
Version:       0.1.2
Warning:       unmaintained
Title:         `cpuid-bool` has been renamed to `cpufeatures`
Date:          2021-05-06
ID:            RUSTSEC-2021-0064
URL:           https://rustsec.org/advisories/RUSTSEC-2021-0064
2021-07-16 15:04:56 +03:00
Dmitry Rodionov
ed0fcfa9b7 replace parse_duration crate because of unpatched known vulnerability
resolves #87
2021-07-16 14:30:27 +03:00
Heikki Linnakangas
c5509b05de Revert accidental change to vendor/postgres.
I accidentally changed it in befefe8d84.
2021-07-16 12:37:10 +03:00
Heikki Linnakangas
befefe8d84 Run 'cargo fmt'.
Fixes a few formatting discrepancies had crept in recently.
2021-07-14 22:03:14 +03:00
Konstantin Knizhnik
ad92b66eed Fix TimestampTz type to i64 to be compatbile with Postgres 2021-07-14 15:55:12 +03:00
Heikki Linnakangas
d119f2bcce Add unit test for branch creation.
This is pretty similar to the python 'test_branch_behind' test, but I
find it useful to have a small unit test for it too.
2021-07-13 09:54:27 +03:00
Dmitry Ivanov
038dc6c629 Symlink Pipfile (& Pipfile.lock) at the top level
When executed, pipenv shell creates a fresh Pipfile if none
is found in the current directory. This is confusing,
hence the patch to symlink it at the top level, which
is a good starting point for various commands.
2021-07-12 21:30:52 +03:00
Heikki Linnakangas
bfc27bee5e Revert the fix to allegedly inaccurate comment.
I misread the code. It does indeed only call checkpoint() every 10
segments. Revert that change, but keep the rest of the comment fixes.
2021-07-10 18:53:47 +03:00
Heikki Linnakangas
19528de03e Fix inaccurate comment on how often we checkpoint the repository. 2021-07-10 18:30:47 +03:00
Konstantin Knizhnik
3e69c41a47 Add XLOG_HEAP_OPMASK to pg_contants 2021-07-10 10:09:56 +03:00
Konstantin Knizhnik
97681acfcf Replace XLR_RMGR_INFO_MASK with XLOG_HEAP_OPMASK 2021-07-10 10:09:56 +03:00
Konstantin Knizhnik
baf8800b96 Fix incorrect mask in wldecoder 2021-07-10 10:09:56 +03:00
Konstantin Knizhnik
577af8a459 Store page image with the same LSN as replaced WAL record 2021-07-09 17:24:25 +03:00
Dmitry Rodionov
75e717fe86 allow both domains and ip addresses in connection options for
pageserver and wal keeper. Also updated PageServerNode definition in
control plane to account for that. resolves #303
2021-07-09 16:46:21 +03:00
Stas Kelvich
4987d5ee1f reduce lodding in wal_acceptor 2021-07-09 16:45:48 +03:00
anastasia
462b8801d2 Fix list_rels visibility check: some relations after initdb may have lsn equal to last_valid_lsn() 2021-07-09 15:25:53 +03:00
Dmitry Ivanov
2712eaee15 [postgres] Enable seccomp bpf 2021-07-09 14:59:45 +03:00
Eric Seppanen
96f4ddd243 cargo fmt 2021-07-05 21:50:42 -07:00
Heikki Linnakangas
3386ce6f35 Introduce common enum ObjectVal for all values stored in ObjectStore.
Based on Konstantin's original patch (PR #275), but I introduced helper
functions for serializing/deserializing the different kinds of
ObjectValues, which made it more pleasant to use, as the deserialization
checks are now performed in the helper functions.
2021-06-30 12:50:35 +03:00
Heikki Linnakangas
9c856ecf43 Update TODO comment now that we handle relation drops
Commit ced338fd20 neglected this comment. In the passing, also move around
the comments explaining the two kinds of thing we store per page.
2021-06-30 12:50:32 +03:00
anastasia
d4329887b3 remove restore_s3.rs that uses obsolete data format 2021-06-29 18:12:32 +03:00
Heikki Linnakangas
ced338fd20 Handle relation DROPs in page server.
Add back code to parse transaction commit and abort records, and in
particular the list of dropped relations in them. Add 'put_unlink'
function to the Timeline trait and implementation. We had the code to
handle dropped relations in the GC code and elsewhere in ObjectRepository
already, but there was nothing to create the RelationSizeEntry::Unlink
tombstone entries until now. Also add a test to check that GC correctly
removes all page versions of a dropped relation.

Implements https://github.com/zenithdb/zenith/issues/232, except for the
"orphaned" rels.

Reviewed-by: Konstantin Knizhnik
2021-06-29 00:27:10 +03:00
Heikki Linnakangas
44c35722d8 Remove a bunch of dead code
Some of these were related to handling various WAL records that are not
related to any relations, like pg_multixact updates. These should have
been removed in the revert commit 6a9c036ac1, but I missed them.

Also, we didn't anything with commit/abort records. We will start
parsing commit/abort records in the next commit, but seems better to
add that from clean slate.

Reviewed-by: Konstantin Knizhnik
2021-06-29 00:26:53 +03:00
Heikki Linnakangas
ec44f4b299 Add test for Garbage Collection.
This expose a command in in page server to run GC immediately on a given
timeline. It's just for testing purposes.
2021-06-28 17:07:28 +03:00
Heikki Linnakangas
a31bba19b0 Compact RocksDB database after garbage collection.
Without this step, the page versions won't actually be removed, they're
just marked for deletion on the next RocksDB "merge" or "compact"
operation.

Author: Konstantin Knizhnik
2021-06-28 16:53:26 +03:00
Heikki Linnakangas
0dd46061a8 Improve logging and comments in GC.
- Print the number of dropped relations, and the number of relations
  encountered overall.

- If a block has only one page version, the latest one, don't count it as
  a "truncated" version history. Only count pages for which we actually
  removed some old versions.

- Change "last" to "latest" in variable names and comments. "Last" could
  be interpreted as "oldest", but here it means "newest".

- Add a comment noting that the GC code depends on get_page_at_lsn_nowait
  to store the materialized page version in the repository.

- Change "last" to "latest" in variable names for clarity. "Last" could
  be interpreted as the oldest, but here it means newest.
2021-06-28 16:53:13 +03:00
Heikki Linnakangas
cb2ddf06d0 Refactor code to do one iteration of GC to separate function. 2021-06-28 16:49:21 +03:00