mirror of
https://github.com/neondatabase/neon.git
synced 2026-05-24 16:40:38 +00:00
ad6f538aef
# Problem On-demand downloads are still using `tokio::fs`, which we know is inefficient. # Changes - Add `pagebench ondemand-download-churn` to quantify on-demand download throughput - Requires dumping layer map, which required making `history_buffer` impl `Deserialize` - Implement an equivalent of `tokio::io::copy_buf` for owned buffers => `owned_buffers_io` module and children. - Make layer file download sensitive to `io_engine::get()`, using VirtualFile + above copy loop - For this, I had to move some code into the `retry_download`, e.g., `sync_all()` call. Drive-by: - fix missing escaping in `scripts/ps_ec2_setup_instance_store` - if we failed in retry_download to create a file, we'd try to remove it, encounter `NotFound`, and `abort()` the process using `on_fatal_io_error`. This PR adds treats `NotFound` as a success. # Testing Functional - The copy loop is generic & unit tested. Performance - Used the `ondemand-download-churn` benchmark to manually test against real S3. - Results (public Notion page): https://neondatabase.notion.site/Benchmarking-tokio-epoll-uring-on-demand-downloads-2024-04-15-newer-code-03c0fdc475c54492b44d9627b6e4e710?pvs=4 - Performance is equivalent at low concurrency. Jumpier situation at high concurrency, but, still less CPU / throughput with tokio-epoll-uring. - It’s a win. # Future Work Turn the manual performance testing described in the above results document into a performance regression test: https://github.com/neondatabase/neon/issues/7146
52 lines
1.5 KiB
Rust
52 lines
1.5 KiB
Rust
use clap::Parser;
|
|
use utils::logging;
|
|
|
|
/// Re-usable pieces of code that aren't CLI-specific.
|
|
mod util {
|
|
pub(crate) mod request_stats;
|
|
#[macro_use]
|
|
pub(crate) mod tokio_thread_local_stats;
|
|
/// Re-usable pieces of CLI-specific code.
|
|
pub(crate) mod cli {
|
|
pub(crate) mod targets;
|
|
}
|
|
}
|
|
|
|
/// The pagebench CLI sub-commands, dispatched in [`main`] below.
|
|
mod cmd {
|
|
pub(super) mod basebackup;
|
|
pub(super) mod getpage_latest_lsn;
|
|
pub(super) mod ondemand_download_churn;
|
|
pub(super) mod trigger_initial_size_calculation;
|
|
}
|
|
|
|
/// Component-level performance test for pageserver.
|
|
#[derive(clap::Parser)]
|
|
enum Args {
|
|
Basebackup(cmd::basebackup::Args),
|
|
GetPageLatestLsn(cmd::getpage_latest_lsn::Args),
|
|
TriggerInitialSizeCalculation(cmd::trigger_initial_size_calculation::Args),
|
|
OndemandDownloadChurn(cmd::ondemand_download_churn::Args),
|
|
}
|
|
|
|
fn main() {
|
|
logging::init(
|
|
logging::LogFormat::Plain,
|
|
logging::TracingErrorLayerEnablement::Disabled,
|
|
logging::Output::Stderr,
|
|
)
|
|
.unwrap();
|
|
logging::replace_panic_hook_with_tracing_panic_hook().forget();
|
|
|
|
let args = Args::parse();
|
|
match args {
|
|
Args::Basebackup(args) => cmd::basebackup::main(args),
|
|
Args::GetPageLatestLsn(args) => cmd::getpage_latest_lsn::main(args),
|
|
Args::TriggerInitialSizeCalculation(args) => {
|
|
cmd::trigger_initial_size_calculation::main(args)
|
|
}
|
|
Args::OndemandDownloadChurn(args) => cmd::ondemand_download_churn::main(args),
|
|
}
|
|
.unwrap()
|
|
}
|