greptimedb/docs/how-to/how-to-profile-memory.md

Profile memory usage of GreptimeDB

This crate provides an easy approach to dump memory profiling info.

Prerequisites

jemalloc

jeprof is already compiled in the target directory of GreptimeDB. You can find the binary and use it.

# find jeprof binary
find . -name 'jeprof'
# add executable permission
chmod +x <path_to_jeprof>

The path is usually under ./target/${PROFILE}/build/tikv-jemalloc-sys-${HASH}/out/build/bin/jeprof. The default version of jemalloc installed from the package manager may not have the --collapsed option. You may need to check the whether the jeprof version is >= 5.3.0 if you want to install it from the package manager.

# for macOS
brew install jemalloc

# for Ubuntu
sudo apt install libjemalloc-dev

flamegraph

curl https://raw.githubusercontent.com/brendangregg/FlameGraph/master/flamegraph.pl > ./flamegraph.pl

Profiling

Start GreptimeDB instance with environment variables:

# for Linux
MALLOC_CONF=prof:true ./target/debug/greptime standalone start

# for macOS
_RJEM_MALLOC_CONF=prof:true ./target/debug/greptime standalone start

Dump memory profiling data through HTTP API:

curl -X POST localhost:4000/debug/prof/mem > greptime.hprof

You can periodically dump profiling data and compare them to find the delta memory usage.

Analyze profiling data with flamegraph

To create flamegraph according to dumped profiling data:

sudo apt install -y libjemalloc-dev

jeprof <path_to_greptime_binary> <profile_data> --collapse | ./flamegraph.pl > mem-prof.svg

jeprof <path_to_greptime_binary> --base <baseline_prof> <profile_data> --collapse | ./flamegraph.pl > output.svg