e386a366d0
* feat/gdump: ### Add Support for Jemalloc Gdump Flag - **`jemalloc.rs`**: Introduced `PROF_GDUMP` constant and added functions `set_gdump_active` and `is_gdump_active` to manage the gdump flag. - **`error.rs`**: Added error handling for reading and updating the jemalloc gdump flag with `ReadGdump` and `UpdateGdump` errors. - **`lib.rs`**: Exposed `is_gdump_active` and `set_gdump_active` functions for non-Windows platforms. - **`http.rs`**: Added HTTP routes for checking and toggling the jemalloc gdump flag status. - **`mem_prof.rs`**: Implemented handlers `gdump_toggle_handler` and `gdump_status_handler` for managing gdump flag via HTTP requests. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * Update docs/how-to/how-to-profile-memory.md Co-authored-by: shuiyisong <113876041+shuiyisong@users.noreply.github.com> * fix: typo in docs Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> --------- Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> Co-authored-by: shuiyisong <113876041+shuiyisong@users.noreply.github.com>
3.2 KiB
Profile memory usage of GreptimeDB
This crate provides an easy approach to dump memory profiling info. A set of ready to use scripts is provided in docs/how-to/memory-profile-scripts.
Prerequisites
jemalloc
jeprof is already compiled in the target directory of GreptimeDB. You can find the binary and use it.
# find jeprof binary
find . -name 'jeprof'
# add executable permission
chmod +x <path_to_jeprof>
The path is usually under ./target/${PROFILE}/build/tikv-jemalloc-sys-${HASH}/out/build/bin/jeprof.
The default version of jemalloc installed from the package manager may not have the --collapsed option.
You may need to check the whether the jeprof version is >= 5.3.0 if you want to install it from the package manager.
# for macOS
brew install jemalloc
# for Ubuntu
sudo apt install libjemalloc-dev
flamegraph
curl https://raw.githubusercontent.com/brendangregg/FlameGraph/master/flamegraph.pl > ./flamegraph.pl
Profiling
Enable memory profiling for greptimedb binary
Start GreptimeDB instance with environment variables:
# for Linux
MALLOC_CONF=prof:true ./target/debug/greptime standalone start
# for macOS
_RJEM_MALLOC_CONF=prof:true ./target/debug/greptime standalone start
Memory profiling for greptimedb docker image
We have memory profiling enabled and activated by default in our official docker image.
This behavior is controlled by configuration enable_heap_profiling:
[memory]
# Whether to enable heap profiling activation during startup.
# Default is true.
enable_heap_profiling = true
To disable memory profiling, set enable_heap_profiling to false.
Memory profiling control
You can control heap profiling activation using the new HTTP APIs:
# Check current profiling status
curl -X GET localhost:4000/debug/prof/mem/status
# Activate heap profiling (if not already active)
curl -X POST localhost:4000/debug/prof/mem/activate
# Deactivate heap profiling
curl -X POST localhost:4000/debug/prof/mem/deactivate
# Activate gdump feature that dumps memory profiling data every time virtual memory usage exceeds previous maximum value.
curl -X POST localhost:4000/debug/prof/mem/gdump -d 'activate=true'
# Deactivate gdump.
curl -X POST localhost:4000/debug/prof/mem/gdump -d 'activate=false'
# Retrieve current gdump status.
curl -X GET localhost:4000/debug/prof/mem/gdump
Dump memory profiling data
Dump memory profiling data through HTTP API:
curl -X POST localhost:4000/debug/prof/mem > greptime.hprof
# or output flamegraph directly
curl -X POST "localhost:4000/debug/prof/mem?output=flamegraph" > greptime.svg
# or output pprof format
curl -X POST "localhost:4000/debug/prof/mem?output=proto" > greptime.pprof
You can periodically dump profiling data and compare them to find the delta memory usage.
Analyze profiling data with flamegraph
To create flamegraph according to dumped profiling data:
sudo apt install -y libjemalloc-dev
jeprof <path_to_greptime_binary> <profile_data> --collapse | ./flamegraph.pl > mem-prof.svg
jeprof <path_to_greptime_binary> --base <baseline_prof> <profile_data> --collapse | ./flamegraph.pl > output.svg