From 21a11822e8fa7119c04c5e95bd40e193d78bcbc8 Mon Sep 17 00:00:00 2001
From: Christian Schwarz <christian@neon.tech>
Date: Mon, 29 Jan 2024 16:21:22 +0000
Subject: [PATCH] results: tokio-epoll-uring 3.3kGetPage/s@240k IOPS, std-fs:
 1.2kGetPage/s@80k IOPS

We have immense read amplification, I think we read the same blk
multiple times during one getpage request.

Before the switch to O_DIRECT, we'd go to the kernel page cache
many times. std-fs has an edge there, it's more efficient than
tokio-epoll-uring for workloads that have a high kernel page cache hit
rate.

With O_DIRECT, we now go to the disk for each read, making the inefficiency apparent.
tokio-epoll-uring is mcuh better there, as we can see it can drive up to
240k IOPS, which is 2GiBs random 8k reads, which afaik is the max that
the EC2 NVMe allows.
CPU isn't near 100%.
SO, we're IO bound.

Idea to try out to reduce the read amplification: request-local page cache.