rust/neon
1
0
Fork 0
mirror of https://github.com/neondatabase/neon.git synced 2026-09-07 05:49:02 +00:00
Code Issues Packages Projects Releases Wiki Activity
6,718 Commits 1,487 Branches 282 Tags
conrad/proxy-postgres-errors
Commit Graph
55 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Patrick Insinger 191d9d2b74 par_fsync - use VirtualFile 2022-01-04 20:40:57 -08:00
Patrick Insinger 24c8dab86f pageserver - parallelize checkpoint fsyncs 2022-01-04 20:40:57 -08:00
Patrick Insinger 298bc588f9 pageserver - don't try to GC InMemoryLayers 2021-11-15 09:01:45 -08:00
Patrick Insinger 1ce4976e36 pageserver - track size of VecMaps 2021-11-10 11:09:34 -08:00
Patrick Insinger 0457fe81a9 pageserver - make PageVersion an enum 2021-11-03 09:28:49 -07:00
Patrick Insinger b532470792 Set SO_REUSEADDR for all TCP listeners 2021-10-29 12:45:26 -07:00
Patrick Insinger 2dde20a227 Bump MSRV to 1.55 2021-10-15 09:10:08 -07:00
Patrick Insinger 1c29de81de pageserver - remove lsn from WALRecord 2021-10-13 00:03:42 -07:00
Patrick Insinger 160c4aff61 pageserver - use write guard for checkpointing 2021-10-12 10:02:15 -07:00
Patrick Insinger 6e5ca5dc5c pageserver - create TimelineWriter 2021-10-12 10:02:15 -07:00
Patrick Insinger 3f0ebc6a40 pageserver - move early File::open call 2021-10-09 08:45:52 -07:00
Patrick Insinger 0baf4bc796 fix cargo doc complaints 2021-10-09 08:45:46 -07:00
Patrick Insinger c356030660 pageserver - use VecMap for delta metadata & sizes 2021-10-08 15:05:22 -07:00
Patrick Insinger c4bb6d78d4 pageserver - use VecMap for in memory segsizes 2021-10-08 14:37:32 -07:00
Patrick Insinger 3b82e806f2 pageserver - use VecMap for in-memory PageVersions 2021-10-08 14:11:07 -07:00
Patrick Insinger b3b8f18f61 tests - fix get_timeline_size signature 2021-10-07 15:38:22 -07:00
Patrick Insinger 9c936034b6 pageserver - fix newer clippy lints 2021-10-05 00:28:14 -07:00
Patrick Insinger d134a9856e pageserver - introduce RepoHarness for testing 2021-10-04 08:36:35 -07:00
Patrick Insinger 664b99b5ac pageserver - use constant TIMELINE_ID for tests 2021-10-04 08:36:35 -07:00
Patrick Insinger 7095a5d551 pageserver - reject and backup future layer files 
If a layer file is found with LSN after the disk_consistent_lsn, it is
renamed (to avoid conflicts with new layer files) and a warning is logged.
 2021-10-01 11:41:39 -07:00
Patrick Insinger 538c2a2a3e pageserver - store timeline metadata durably 
The metadata file is now always 512 bytes. The last 4 bytes are a
crc32c checksum of the previous 508 bytes. Padding zeroes are added
between the serde serialization and the start of the checksum.

A single write call is used, and the file is fsyncd after.
On file creation, the parent directory is fsyncd as well.
 2021-10-01 11:41:39 -07:00
Patrick Insinger 62f83869f1 pageserver - fsync image/delta layers 
Ensure image and delta layer files are durable.
Also, fsync the parent directory to ensure the directory entries are
durable.
 2021-10-01 11:41:39 -07:00
Patrick Insinger 69670b61c4 pageserver - use crashsafe_dir utility 
Replace usage of std::fs::create_dir/create_dir_all with crashsafe
equivalents.
 2021-10-01 11:41:39 -07:00
Patrick Insinger 0a8aaa2c24 zenith_utils - add crashsafe_dir 
Utility for creating directories and directory trees in a crash safe
manor.

Minimizes calls to fsync for trees.
 2021-10-01 11:41:39 -07:00
Patrick Insinger 7db3a9e7d9 walredo - don't use RefCell on stdin/stdout 2021-09-23 08:42:58 -07:00
Patrick Insinger c81ee3bd5b Add some comments to the checkpoint process 2021-09-23 13:19:45 +03:00
Patrick Insinger ea4c3639e3 Include layer metadata in layer summary chapters 
Include all data stored in layer filenames and the tenant+timeline IDs
inside a summary chapter. Use this chapter in the `dump_layerfile`
utility.
 2021-09-20 07:57:51 -07:00
Patrick Insinger 25b7d424ab Prevent frozen InMemoryLayer races 
Instead of panicking when a race happens, retry the operation after
getting a new layer.
 2021-09-15 20:50:51 -07:00
Patrick Insinger a5bd306db9 Ensure InMemoryLayer predecessor updated correctly 
When the new open InMemoryLayer predecessor is updated, ensure it was
pointing to the old frozen layer.
 2021-09-15 16:04:49 -07:00
Patrick Insinger 0cbee4a416 Don't hold lock on LayerMap while writing to disk 2021-09-15 16:04:49 -07:00
Patrick Insinger 91ff09151d Remove disk IO from InMemoryLayer::freeze 
Move the creation of Image and Delta layers from
`InMemoryLayer::freeze()` to `InMemoryLayer::write_to_disk`.
 2021-09-15 16:04:49 -07:00
Patrick Insinger fea5954b18 Change filling gap println! to trace! 2021-09-15 14:22:04 -07:00
Patrick Insinger d150f3ce8c Detect writes on frozen InMemoryLayers 
Data written to frozen layers is lost. It will not appear in on-disk
structures or in successor InMemoryLayers. Here we detect this race, and
fail. I think this race is rare, but this should make it easier to track
down when it happens.
 2021-09-14 11:44:48 -07:00
Patrick Insinger cff4572774 Avoid race in get_layer_for_write 
Implement the changes suggested in a comment, create
`get_layer_for_read_locked` so that `get_layer_for_write` doesn't have
to drop the LayerMap lock when searching for the predecessor.
 2021-09-14 11:24:24 -07:00
Patrick Insinger 7c62a57e54 initialize tenant_mgr after daemonizing 
Ran into problems launching the WAL redo process on OS X after 4b73ad.
Launching the `initdb` process was met with "bad file descriptor" errors.
Using dtrace, I found shortly after calling `posix_spawn` for `initdb`,
`kevent` was returning this error.

I haven't dug super deep to see if the daemonization itself is the
problem, but this commit fixes it for me. My hunch is that some file
descriptors used when the Tokio runtime is initailzed become invalid
in the daemon process.
 2021-09-10 13:00:39 +03:00
Patrick Insinger 1b9e49eb60 pageserver - update unload() comment 
Update comment to reflect changes made in 5ac4a2 and 98f496
 2021-09-07 08:19:42 -07:00
Patrick Insinger 7507f4b309 zenith_utils - box BidiStream::Tls variant 
Clippy warns that one variant is 40 bytes and the other is 568 bytes.
Box the larger variant to avoid this warning
 2021-09-02 09:16:03 -07:00
Patrick Insinger 5ac4a27042 image_layer - read images directly from disk 
Avoid slurping entire image files into memory.

For blocky segments, we write the bytes directly to a bookfile chapter.
The blocks are a fixed size, which allows for random access.
 2021-08-30 10:34:36 -07:00
Patrick Insinger 7c7e89e2ea layered_repo - atomic last/prev record_lsn 
Make a new type that stores both Lsns. Use an RwLock for thread safety.
 2021-08-30 09:40:13 -07:00
Patrick Insinger 561bf2c510 circleci - fix test summary 2021-08-30 09:18:49 -07:00
Patrick Insinger 98f49671c1 delta_layer - read page versions from disk 
split the page versions into two chapters:
PAGE_VERSION_METAS - a rust BTreeMap from (block #, lsn) -> page & WAL
byte ranges in PAGE_VERSIONS_CHAPTER
PAGE_VERSIONS_CHAPTER - raw page images and serialized WAL records
 2021-08-30 09:12:38 -07:00
Patrick Insinger d265b4cdd3 waldecoder - check for trailing bytes 
When we parse the main data in a WAL record, ensure we consume all bytes.
 2021-08-26 10:24:33 -07:00
Patrick Insinger cc169a6896 pageserver - config file 
To simplify cloud ops, allow configuration via file.
toml is used as the config format, and the file is stored in the working
directory.
Arguments used at initialization are saved in the config file.
Config file params may be overridden by CLI arguments.
 2021-06-14 09:40:22 -07:00
Patrick Insinger 77366b7a76 pageserver - remove env variables 
Use CLI args instead of environment variables to parameterize the
working directory and postgres distirbution.

Before this change, there was a mixture of environment variables and CLI
arguments that needed to be set. Moving to a single input simplifies
cloud configuration management.
 2021-06-14 09:40:22 -07:00
Patrick Insinger 47694ea4f5 zenith push 2021-06-02 17:20:49 -04:00
Patrick Insinger 3364a8d442 pageserver - timeline history api 2021-06-02 16:20:26 -04:00
Patrick Insinger 23be5021f8 Remote CLI command 2021-05-18 01:04:00 -04:00
Patrick Insinger f954d5c501 pageserver - separate pagestream messages 2021-05-17 17:17:08 -04:00
Patrick Insinger 53ea6702bd zenith - pg list relax pageserver dependency 2021-05-17 11:14:01 -04:00
Patrick Insinger 4c5e23d014 pageserver - fix ParameterStatus write call 2021-05-12 20:59:04 -04:00
Patrick Insinger 99d80aba52 use pageserver for pg list command 2021-05-12 12:34:03 +03:00
Patrick Insinger 372617a4f5 test_runner - pgrep remove -c arg 
macOS doesn't support it
 2021-05-11 17:52:22 -04:00
Patrick Insinger 49d1921a28 page_server - add python api tests 2021-05-11 14:16:22 -04:00
Patrick Insinger d8e509d29e page_service - use anyhow for error handling 2021-05-11 14:11:10 -04:00
Patrick Insinger d5bfe84d9e cargo fmt 2021-05-11 12:35:09 -04:00