Commit Graph
729 Commits
Author SHA1 Message Date
Bojan Serafimov 98d8d65c83 Cleanup 2022-05-24 14:03:07 -04:00
Bojan Serafimov 1e9b628c25 Add period 2022-05-24 12:49:30 -04:00
Bojan Serafimov 31dc9e6abd Use queue 2022-05-24 10:50:56 -04:00
Bojan Serafimov 0d5fee3ab7 Add gc pool 2022-05-23 23:42:25 -04:00
Bojan Serafimov dc47f9ccf1 Simplify 2022-05-23 23:18:25 -04:00
Bojan Serafimov fefbff8981 Add job status 2022-05-23 22:44:48 -04:00
Bojan Serafimov eefc7aa792 Add todo 2022-05-23 22:33:53 -04:00
Bojan Serafimov f3c71899be Add recurring job 2022-05-23 22:32:14 -04:00
Bojan Serafimov de55b2f139 Handle panic 2022-05-22 16:12:00 -04:00
Bojan Serafimov 5a19ac02c9 Add todo 2022-05-22 15:53:52 -04:00
Bojan Serafimov 13b374a2df Add shutdown command 2022-05-22 12:34:53 -04:00
Bojan Serafimov b9814222f9 WIP 2022-05-12 12:28:46 -04:00
Kirill Bulatov 0a7735a656 Rework remote storage sync queue, general refactoring 2022-05-07 01:33:33 +03:00
Kirill Bulatov 64a602b8f3 Delete timeline layers 2022-05-07 01:33:33 +03:00
Kirill Bulatov 10e4da3997 Rework timeline batching 2022-05-07 01:33:33 +03:00
Kirill Bulatov de37f982db Share the remote storage as a crate 2022-05-07 00:30:36 +03:00
Kirill Bulatov 2ef0e5c6ed Do not require metadata in every upload sync task 2022-05-05 18:26:39 +03:00
Kirill Bulatov 52a7e3155e Add local path to the Layer trait and historic layers 2022-05-05 18:26:39 +03:00
Dmitry Rodionov 0f3ec83172 avoid detach with alive branches 2022-05-05 12:54:42 +03:00
bojanserafimov bc569dde51 Remove some unwraps from waldecoder (#1539) 2022-05-04 17:41:05 -04:00
Anastasia Lubennikova e2cf77441d Implement pg_database_size().
In this implementation dbsize equals sum of all relation sizes, excluding shared ones.
2022-05-04 18:14:45 +03:00
Stas Kelvich 5642d0b2b8 Change shutdown_process_on_error thread spawn settings.
Now princeple is following: acceptor threads (libpq and http) error will
bring the pageserver down, but all per-tenant thread failures will be treated
as an error.
2022-05-04 00:42:57 +03:00
Dmitry Rodionov 2f83f793bc print more details when thread fails 2022-05-03 18:31:23 +03:00
Anastasia Lubennikova 2f9b17b9e5 Add simple test of pageserver recovery after crash. To cause a crash, use failpoints in checkpointer 2022-05-03 17:13:09 +03:00
Dmitry Rodionov e7cba0b607 use thiserror instead of anyhow in disk_btree 2022-05-03 15:34:23 +03:00
Dmitry Rodionov ff7e9a86c6 turn panic into an error with more details 2022-05-03 12:44:42 +03:00
Heikki Linnakangas 9ede38b6c4 Support finding LSN from a commit timestamp.
A new `get_lsn_by_timestamp` command is added to the libpq page service
API.

An extra timestamp field is now stored in an extra field after each
Clog page. It is the timestamp of the latest commit, among all the
transactions on the Clog page. To find the overall latest commit, we
need to scan all Clog pages, but this isn't a very frequent operation
so that's not too bad.

To find the LSN that corresponds to a timestamp, we perform a binary
search. The binary search starts with min = last LSN when GC ran, and
max = latest LSN on the timeline. On each iteration of the search we
check if there are any commits with a higher-than-requested timestamp
at that LSN.

Implements github issue 1361.
2022-05-03 09:28:57 +03:00
Konstantin KnizhnikandHeikki Linnakangas baa59512b8 Traverse frozen layer in get_reconstruct_data in reverse order (#1601)
* Traverse frozen layer in get_reconstruct_data in reverse order

* Fix comments on frozen layers.

Note explicitly the order that the layers are in the queue.

* Add fail point to reproduce failpoint iteration error

Co-authored-by: Heikki Linnakangas <heikki@neon.tech>
2022-05-03 08:07:14 +03:00
Dmitry Rodionov ad25736f3a Exit pageserver process with correct error code
When we shutdown pageserver due to an error (e g one of th important
thrads panicked) use 1 exit code so systemd can properly restart it
2022-05-02 19:04:45 +03:00
Dhammika Pathirana f3f12db2cb Add gc churn threshold knob (#1594)
Signed-off-by: Dhammika Pathirana <dhammika@gmail.com>
2022-05-01 13:13:17 -07:00
Kirill Bulatov 7e1db8c8a1 Show which virtual file got the deserialization errors 2022-04-29 21:40:57 +03:00
Dmitry Rodionov 05f8e6a050 Use fsync+rename for atomic downloads from remote storage
Use failpoint in test_remote_storage to check the behavior
2022-04-29 15:53:56 +03:00
Kirill Bulatov 2911eb084a Remove timeline files on detach 2022-04-29 09:19:18 +03:00
Kirill Bulatov 6cca57f95a Properly remove from the local timeline map 2022-04-29 09:19:18 +03:00
Kirill Bulatov 4a46b01caf Properly populate local timeline map 2022-04-29 09:19:18 +03:00
Anastasia Lubennikova 5c5c3c64f3 Fix tenant config parsing. Add a test 2022-04-28 11:49:19 +03:00
Dhammika Pathirana aeb4f81c3b Add branch traversal unit test
Signed-off-by: Dhammika Pathirana <dhammika@gmail.com>
2022-04-27 00:05:13 -07:00
Dhammika Pathirana b2e35fffa6 Fix ancestor layer traversal (#1484)
Signed-off-by: Dhammika Pathirana <dhammika@gmail.com>
2022-04-27 00:05:13 -07:00
Kirill Bulatov 778744d35c Limit concurrent S3 and IAM interactions 2022-04-26 13:49:37 +03:00
Dmitry Rodionov eabf6f89e4 Use item.get for tenant config toml parsing
Previously we've used table interface, but there was no easy way to pass
it as an override to pageserver through cli. Use the same strategy as
for remote storage config parsing
2022-04-26 10:15:19 +03:00
Kirill Bulatov fec050ce97 Fix macos clippy issues 2022-04-25 16:23:34 +03:00
Kirill Bulatov 8f6a161271 Show better layer load errors 2022-04-25 14:54:39 +03:00
Heikki Linnakangas 1fb3d08185 Use a 1-byte length header for short blobs.
Notably, this shaves 3 bytes from each small WAL record stored in
ephemeral or delta layers.
2022-04-22 21:31:27 +03:00
Dmitry Ivanov d3f356e7a8 Update rust-postgres project-wide (#1525)
* Update `rust-postgres` project-wide

This commit points to https://github.com/neondatabase/rust-postgres/commits/neon
in order to test our patches on top of the latest version of this crate.

* [proxy] Update `hmac` and `sha2`
2022-04-22 17:31:58 +03:00
Konstantin Knizhnik 5f83c9290b Make it possible to specify per-tenant configuration parameters
Add tenant config API and 'zenith tenant config' CLI command.
Add 'show' query to pageserver protocol for tenantspecific config parameters

Refactoring: move tenant_config code to a separate module.
Save tenant conf file to tenant's directory, when tenant is created to recover it on pageserver restart.
Ignore error during tenant config loading, while it is not supported by console

Define PiTR interval for GC.

refer #1320
2022-04-22 11:24:29 +03:00
Heikki Linnakangas a4700c9bbe Use pprof to get flamegraph of get_page and get_relsize requests.
This depends on a hacked version of the 'pprof-rs' crate. Because of
that, it's under an optional 'profiling' feature. It is disabled by
default, but enabled for release builds in CircleCI config. It doesn't
currently work on macOS.

The flamegraph is written to 'flamegraph.svg' in the pageserver
workdir when the 'pageserver' process exits.

Add a performance test that runs the perf_pgbench test, with profiling
enabled.
2022-04-21 20:32:48 +03:00
Heikki Linnakangas dafdf9b952 Handle EINTR 2022-04-21 16:37:36 +03:00
Heikki Linnakangas 263d60f12d Add prometheus metric for time spent waiting for WAL to arrive 2022-04-21 16:37:32 +03:00
Kirill Bulatov 81cad6277a Move and library crates into a dedicated directory and rename them 2022-04-21 13:30:33 +03:00
Heikki Linnakangas 9d3779c124 Add a counter for materialized page cache hits. 2022-04-20 21:26:03 +03:00