From 82c30ac7578a8a43338d01ab4322d91974a25e37 Mon Sep 17 00:00:00 2001 From: Christian Schwarz Date: Mon, 8 Jul 2024 09:28:29 +0000 Subject: [PATCH] summary, terminology, history --- docs/rfcs/direct-io-for-reads.md | 83 +++++++++++++++++++++++++++++++- 1 file changed, 82 insertions(+), 1 deletion(-) diff --git a/docs/rfcs/direct-io-for-reads.md b/docs/rfcs/direct-io-for-reads.md index 9691d9e55e..9ffa9e8847 100644 --- a/docs/rfcs/direct-io-for-reads.md +++ b/docs/rfcs/direct-io-for-reads.md @@ -1 +1,82 @@ -Placeholder. Subscribe to the PR to get notified when there's something to read here. +# Direct IO For Pageserver + +## Summary + +This document is a proposal and implementation plan for direct IO in Pageserver. + +## Terminology + +**kernel page cache**: the kernel's page cache is a write-back cache for filesystem contents. +The cached unit is memory-page-sized & aligned chunks of the files that are being cached (typically 4k). +The cache lives in kernel memory and is not directly accessible through userspace. + +**Buffered IO**: the application's read/write system calls go through the kernel page cache. +For example, a 10 byte sized read or write to offset 5000 in a file will load the file contents +at offset `[4096,8192)` into a free page in the kernel page cache. If necessary, it will evict +other pages to make room (cf eviction). Then, the kernel performs a memory-to-memory copy of 10 bytes +from/to the offset `4` (`5000 = 4096 + 4`) within the cached page. If it's a write, the kernel keeps +track of the fact that the page is now "dirty" in some ancillary structure. + +**Writeback**: a buffered read/write syscall returns after the memory-to-memory copy. The moficiations +made by e.g. write system calls are not even *issued* to disk, let alone durable. Instead, the kernel +asynchronously writes back dirtied pages based on a variety of conditions. For us, the most relevant +ones are a) explicit request by userspace (`fsync`) and b) memory pressure. + +**Memory pressure**: the kernel page cache is a best effort service and a user of spare memory capacity. +The kernel page allocator will take pages used by page cache if there is no other free memory available. +Before reusing a page like that, the page has to be written back (writeback, see above). +The far-reaching consequence of this is that **any allocation of anonymous memory can do IO** if the only +way to get that memory is by eviction & re-using a dirty page cache page. +Notably, this includes a simple `malloc` in userspace, because eventually that boils down to `mmap(..., MAP_ANON, ...)`. + +**Direct IO** allows application's read/write system calls to bypass the kernel page cache. The filesystem +is still involved because it is ultimately in charge of mapping the concept of files & offsets within them +to sectors on block devices. + +**"buffered" vs "direct"**: the central distinction between buffered and direct IO is about who allocates and +fills the IO buffers, and who controls when exactly the IOs are issued. In buffered IO, it's the syscall handlers, +kernel page cache, and memory management subsystems (cf "writeback"). In direct IO, all of it is done by +the application. +It takes more effort by the application to program with direct instead of buffered IO. +The return is precise control over and a clear distinction between consumption/modification of memory vs disk. + +**Pageserver PageCache**: Pageserver has an additioanl `PageCache` (referred to as PageCache from here on, as opposed to "kernel page cache"). +Its caching unit is 8KiB which is the Postgres page size. +Currently, it is tiny (128MiB), very much like Postgres's `shared_buffers`. +A miss in PageCache is filled from the filesystem using buffered IO, issued through the `VirtualFile` layer in Pageserver. + +**VirtualFIle** is Pageserver's abstraction for file IO, very similar to the faciltiy in Postgres that bears the same name. +Its historical purpose appears to be working around open file descriptor limitations, which is practically irrelevant on Linux. +However, the faciltiy in Pageserver is useful as an intermediary layer for metrics and abstracts over the different kinds of +IO engines that Pageserver supports (`std-fs` vs `tokio-epoll-uring`). + +## History Of Caching In Pageserver + +For multiple years, Pageserver's `PageCache` was used for all data path read _and write_ IO. +It performed write-back to the kernel using buffered IO. + +We converted it into a read-only cache of immutable data in [PR 4994](https://github.com/neondatabase/neon/pull/4994). + +The introduction of `tokio-epoll-uring` required converting the code base to used owned IO buffers. +The `PageCache` pages are usable as owned IO buffers. + +We then introduced vectored `Timeline::get` (cf RFC 30). +The implementation bypasses PS `PageCache` for delta and image layer data block reads. +(The disk btree embedded in delta & image layers is still `PageCache`'d). + +Most recently, and still ongoing, is [Epic: Bypass PageCache for user data blocks #7386](https://github.com/neondatabase/neon/issues/7386). +The goal there is to eliminate the remaining caching of user data blocks in PS `PageCache`. + +The outcome of the above will be that +1. all data blocks are read through VirtualFile and +2. all indirect blocks (=disk btree blocks) are cached in the PS `PageCache`. +The norm will be very low baseline replacement rates in PS `PageCache`. +High baseline replacement rates will be treated as a signal of resource exhaustion (page cache insufficient to host working set of the PS). +It will be remediated by the storage controller, migrating tenants away to relieve pressure. + +In the future, we may elminate the `PageCache` even for indirect blocks. +For example with an LRU cache that has as unit the entire disk btree content +instead of individual blocks. + +## +