Bojan Serafimov
32b4e0fda2
Fix
2022-05-24 15:14:33 -04:00
Bojan Serafimov
6b4e17ef89
Start workers
2022-05-24 15:03:18 -04:00
Bojan Serafimov
07e6dd809d
Start jobs when tenant activates
2022-05-24 14:25:15 -04:00
Bojan Serafimov
b028f12b06
Fix
2022-05-24 14:14:29 -04:00
Bojan Serafimov
3975417cae
Add metadata
2022-05-24 14:12:53 -04:00
Bojan Serafimov
98d8d65c83
Cleanup
2022-05-24 14:03:07 -04:00
Bojan Serafimov
1e9b628c25
Add period
2022-05-24 12:49:30 -04:00
Bojan Serafimov
31dc9e6abd
Use queue
2022-05-24 10:50:56 -04:00
Bojan Serafimov
0d5fee3ab7
Add gc pool
2022-05-23 23:42:25 -04:00
Bojan Serafimov
dc47f9ccf1
Simplify
2022-05-23 23:18:25 -04:00
Bojan Serafimov
fefbff8981
Add job status
2022-05-23 22:44:48 -04:00
Bojan Serafimov
eefc7aa792
Add todo
2022-05-23 22:33:53 -04:00
Bojan Serafimov
f3c71899be
Add recurring job
2022-05-23 22:32:14 -04:00
Bojan Serafimov
de55b2f139
Handle panic
2022-05-22 16:12:00 -04:00
Bojan Serafimov
5a19ac02c9
Add todo
2022-05-22 15:53:52 -04:00
Bojan Serafimov
13b374a2df
Add shutdown command
2022-05-22 12:34:53 -04:00
Bojan Serafimov
b9814222f9
WIP
2022-05-12 12:28:46 -04:00
Kirill Bulatov
0a7735a656
Rework remote storage sync queue, general refactoring
2022-05-07 01:33:33 +03:00
Kirill Bulatov
64a602b8f3
Delete timeline layers
2022-05-07 01:33:33 +03:00
Kirill Bulatov
10e4da3997
Rework timeline batching
2022-05-07 01:33:33 +03:00
Kirill Bulatov
de37f982db
Share the remote storage as a crate
2022-05-07 00:30:36 +03:00
Kirill Bulatov
2ef0e5c6ed
Do not require metadata in every upload sync task
2022-05-05 18:26:39 +03:00
Kirill Bulatov
52a7e3155e
Add local path to the Layer trait and historic layers
2022-05-05 18:26:39 +03:00
Dmitry Rodionov
0f3ec83172
avoid detach with alive branches
2022-05-05 12:54:42 +03:00
bojanserafimov
bc569dde51
Remove some unwraps from waldecoder ( #1539 )
2022-05-04 17:41:05 -04:00
Anastasia Lubennikova
e2cf77441d
Implement pg_database_size().
...
In this implementation dbsize equals sum of all relation sizes, excluding shared ones.
2022-05-04 18:14:45 +03:00
Stas Kelvich
5642d0b2b8
Change shutdown_process_on_error thread spawn settings.
...
Now princeple is following: acceptor threads (libpq and http) error will
bring the pageserver down, but all per-tenant thread failures will be treated
as an error.
2022-05-04 00:42:57 +03:00
Dmitry Rodionov
2f83f793bc
print more details when thread fails
2022-05-03 18:31:23 +03:00
Anastasia Lubennikova
2f9b17b9e5
Add simple test of pageserver recovery after crash. To cause a crash, use failpoints in checkpointer
2022-05-03 17:13:09 +03:00
Dmitry Rodionov
e7cba0b607
use thiserror instead of anyhow in disk_btree
2022-05-03 15:34:23 +03:00
Dmitry Rodionov
ff7e9a86c6
turn panic into an error with more details
2022-05-03 12:44:42 +03:00
Heikki Linnakangas
9ede38b6c4
Support finding LSN from a commit timestamp.
...
A new `get_lsn_by_timestamp` command is added to the libpq page service
API.
An extra timestamp field is now stored in an extra field after each
Clog page. It is the timestamp of the latest commit, among all the
transactions on the Clog page. To find the overall latest commit, we
need to scan all Clog pages, but this isn't a very frequent operation
so that's not too bad.
To find the LSN that corresponds to a timestamp, we perform a binary
search. The binary search starts with min = last LSN when GC ran, and
max = latest LSN on the timeline. On each iteration of the search we
check if there are any commits with a higher-than-requested timestamp
at that LSN.
Implements github issue 1361.
2022-05-03 09:28:57 +03:00
Konstantin Knizhnik
baa59512b8
Traverse frozen layer in get_reconstruct_data in reverse order ( #1601 )
...
* Traverse frozen layer in get_reconstruct_data in reverse order
* Fix comments on frozen layers.
Note explicitly the order that the layers are in the queue.
* Add fail point to reproduce failpoint iteration error
Co-authored-by: Heikki Linnakangas <heikki@neon.tech >
2022-05-03 08:07:14 +03:00
Dmitry Rodionov
ad25736f3a
Exit pageserver process with correct error code
...
When we shutdown pageserver due to an error (e g one of th important
thrads panicked) use 1 exit code so systemd can properly restart it
2022-05-02 19:04:45 +03:00
Dhammika Pathirana
f3f12db2cb
Add gc churn threshold knob ( #1594 )
...
Signed-off-by: Dhammika Pathirana <dhammika@gmail.com >
2022-05-01 13:13:17 -07:00
Kirill Bulatov
7e1db8c8a1
Show which virtual file got the deserialization errors
2022-04-29 21:40:57 +03:00
Dmitry Rodionov
05f8e6a050
Use fsync+rename for atomic downloads from remote storage
...
Use failpoint in test_remote_storage to check the behavior
2022-04-29 15:53:56 +03:00
Kirill Bulatov
2911eb084a
Remove timeline files on detach
2022-04-29 09:19:18 +03:00
Kirill Bulatov
6cca57f95a
Properly remove from the local timeline map
2022-04-29 09:19:18 +03:00
Kirill Bulatov
4a46b01caf
Properly populate local timeline map
2022-04-29 09:19:18 +03:00
Anastasia Lubennikova
5c5c3c64f3
Fix tenant config parsing. Add a test
2022-04-28 11:49:19 +03:00
Dhammika Pathirana
aeb4f81c3b
Add branch traversal unit test
...
Signed-off-by: Dhammika Pathirana <dhammika@gmail.com >
2022-04-27 00:05:13 -07:00
Dhammika Pathirana
b2e35fffa6
Fix ancestor layer traversal ( #1484 )
...
Signed-off-by: Dhammika Pathirana <dhammika@gmail.com >
2022-04-27 00:05:13 -07:00
Kirill Bulatov
778744d35c
Limit concurrent S3 and IAM interactions
2022-04-26 13:49:37 +03:00
Dmitry Rodionov
eabf6f89e4
Use item.get for tenant config toml parsing
...
Previously we've used table interface, but there was no easy way to pass
it as an override to pageserver through cli. Use the same strategy as
for remote storage config parsing
2022-04-26 10:15:19 +03:00
Kirill Bulatov
fec050ce97
Fix macos clippy issues
2022-04-25 16:23:34 +03:00
Kirill Bulatov
8f6a161271
Show better layer load errors
2022-04-25 14:54:39 +03:00
Heikki Linnakangas
1fb3d08185
Use a 1-byte length header for short blobs.
...
Notably, this shaves 3 bytes from each small WAL record stored in
ephemeral or delta layers.
2022-04-22 21:31:27 +03:00
Dmitry Ivanov
d3f356e7a8
Update rust-postgres project-wide ( #1525 )
...
* Update `rust-postgres` project-wide
This commit points to https://github.com/neondatabase/rust-postgres/commits/neon
in order to test our patches on top of the latest version of this crate.
* [proxy] Update `hmac` and `sha2`
2022-04-22 17:31:58 +03:00
Konstantin Knizhnik
5f83c9290b
Make it possible to specify per-tenant configuration parameters
...
Add tenant config API and 'zenith tenant config' CLI command.
Add 'show' query to pageserver protocol for tenantspecific config parameters
Refactoring: move tenant_config code to a separate module.
Save tenant conf file to tenant's directory, when tenant is created to recover it on pageserver restart.
Ignore error during tenant config loading, while it is not supported by console
Define PiTR interval for GC.
refer #1320
2022-04-22 11:24:29 +03:00
