neon/benches at dcb24ce170573a2ae6ed29467669d03c73b589e6 - neon

rust/neon

mirror of https://github.com/neondatabase/neon.git synced 2026-01-08 05:52:55 +00:00

Files

Erik Grinaker dcb24ce170 safekeeper,pageserver: add heap profiling (#9778 )

## Problem

We don't have good observability for memory usage. This would be useful
e.g. to debug OOM incidents or optimize performance or resource usage.

We would also like to use continuous profiling with e.g. [Grafana Cloud
Profiles](https://grafana.com/products/cloud/profiles-for-continuous-profiling/)
(see https://github.com/neondatabase/cloud/issues/14888).

This PR is intended as a proof of concept, to try it out in staging and
drive further discussions about profiling more broadly.

Touches https://github.com/neondatabase/neon/issues/9534.
Touches https://github.com/neondatabase/cloud/issues/14888.
Depends on #9779.
Depends on #9780.

## Summary of changes

Adds a HTTP route `/profile/heap` that takes a heap profile and returns
it. Query parameters:

* `format`: output format (`jemalloc` or `pprof`; default `pprof`).

Unlike CPU profiles (see #9764), heap profiles are not symbolized and
require the original binary to translate addresses to function names. To
make this work with Grafana, we'll probably have to symbolize the
process server-side -- this is left as future work, as is other output
formats like SVG.

Heap profiles don't work on macOS due to limitations in jemalloc.

2024-12-03 11:35:59 +00:00

benchutils.rs

safekeeper: add basic WAL ingestion benchmarks (#9531 )

2024-11-07 13:24:03 +00:00

README.md

safekeeper,pageserver: add CPU profiling (#9764 )

2024-11-21 18:59:46 +00:00

receive_wal.rs

safekeeper,pageserver: add heap profiling (#9778 )

2024-12-03 11:35:59 +00:00

README.md

Safekeeper Benchmarks

To run benchmarks:

# All benchmarks.
cargo bench --package safekeeper

# Specific file.
cargo bench --package safekeeper --bench receive_wal

# Specific benchmark.
cargo bench --package safekeeper --bench receive_wal process_msg/fsync=false

# List available benchmarks.
cargo bench --package safekeeper --benches -- --list

# Generate flamegraph profiles using pprof-rs, profiling for 10 seconds.
# Output in target/criterion/*/profile/flamegraph.svg.
cargo bench --package safekeeper --bench receive_wal process_msg/fsync=false --profile-time 10

Additional charts and statistics are available in target/criterion/report/index.html.

Benchmarks are automatically compared against the previous run. To compare against other runs, see --baseline and --save-baseline.