greptimedb/docs/parquet-tools.md

Parquet Development Tools

GreptimeDB provides the following datanode CLI commands for inspecting, rewriting, benchmarking, and replacing Parquet SST files:

• parquet-meta: inspect a local Parquet file.
• parquet-rewrite: rewrite a local Parquet file with different writer properties.
• parquetbench: benchmark reads from a local GreptimeDB SST or an SST in a configured object store.
• sst-replace: replace an existing Mito SST object and update its manifest metadata.

These are development and recovery tools, not stable user-facing interfaces. Always test a workflow on disposable data before using it on an important region.

Build

parquet-meta, parquet-rewrite, and sst-replace require the dev-tools feature. The same build also includes parquetbench:

cargo build -p cmd --bin greptime --features dev-tools

The examples below use:

GREPTIME=./target/debug/greptime

Run $GREPTIME datanode <COMMAND> --help for the authoritative argument list.

Inspect a Parquet file

parquet-meta reads a local Parquet footer and optional page indexes. It reports file, row-group, and column metadata, including compression, encodings, sizes, page offsets, statistics presence, and bloom-filter/index offsets.

Text output:

$GREPTIME datanode parquet-meta \
  --input /tmp/source.parquet

Machine-readable JSON output:

$GREPTIME datanode parquet-meta \
  --input /tmp/source.parquet \
  --format json > /tmp/source-meta.json

--format accepts text (the default) or json. This command can inspect a general Parquet file; it does not require GreptimeDB SST key-value metadata.

Rewrite a Parquet file

parquet-rewrite has two modes. First, dump a TOML properties file inferred from an existing file:

$GREPTIME datanode parquet-rewrite \
  --input /tmp/source.parquet \
  --dump-properties /tmp/writer-properties.toml

Review and edit the generated file. Its shape is:

[writer]
dictionary_enabled = true
compression = "zstd"
max_row_group_row_count = 8192

[[columns]]
path = ["host"]
dictionary_enabled = true
compression = "zstd"
encoding = "plain"

Supported compression names are uncompressed, snappy, gzip, lzo, brotli, lz4, zstd, and lz4-raw. Supported encodings are plain, delta-binary-packed, delta-length-byte-array, delta-byte-array, and byte-stream-split.

The [writer] table also accepts compression_level, data_page_size_limit, data_page_row_count_limit, and dictionary_page_size_limit. It also accepts column_index_truncate_length and statistics_truncate_length; set either to zero to disable that truncation. A [[columns]] entry overrides dictionary, compression, compression level, or encoding for its column path. Unknown TOML fields are rejected. Compression levels are supported for gzip, brotli, and zstd. A column-level compression_level without compression inherits the writer compression codec; it is rejected when there is no writer codec to inherit.

Rewrite the data using the edited properties:

$GREPTIME datanode parquet-rewrite \
  --input /tmp/source.parquet \
  --properties /tmp/writer-properties.toml \
  --output /tmp/rewritten.parquet

Use --batch-size <ROWS> to control reader batch size. Output and dumped properties files are not replaced unless --overwrite is passed. The output or dump path must not be exactly the same path as the input, even with --overwrite. This check does not resolve symbolic links, hard links, or other spellings of the same path.

The rewrite decodes and writes the Arrow record batches and copies the Parquet key-value metadata, but it creates a new physical Parquet layout. Inspect and validate the result before using it as an SST:

$GREPTIME datanode parquet-meta \
  --input /tmp/rewritten.parquet

The dumped properties are inferred primarily from the first row group. Review them when the source uses different properties across row groups.

Benchmark an SST

parquetbench expects GreptimeDB region metadata embedded in the SST. It can read a local file with the direct reader:

$GREPTIME datanode parquetbench \
  --file-path /tmp/rewritten.parquet \
  --reader direct \
  --iterations 5 \
  --batch-size 8192

Local-file mode cannot be combined with --config, --region-id, --table-dir, or --file-id, and it does not support the flat-prune reader.

To benchmark an SST in the object store configured for a datanode or standalone deployment:

$GREPTIME datanode parquetbench \
  --config /path/to/datanode.toml \
  --region-id 1024:0 \
  --table-dir data/greptime/public/1024 \
  --file-id 00020380-009c-426d-953e-b4e34c15af34 \
  --path-type bare \
  --reader flat-prune \
  --iterations 5

Region mode requires all four of --config, --region-id, --table-dir, and --file-id. --region-id accepts either the packed unsigned integer or <table-id>:<region-number>. --path-type accepts bare, data, or metadata and defaults to bare.

An optional scan configuration selects columns and row groups:

{
  "projection_names": ["host", "value", "ts"],
  "row_groups": [0, 2]
}

$GREPTIME datanode parquetbench \
  --file-path /tmp/rewritten.parquet \
  --scan-config /tmp/parquet-scan.json \
  --iterations 5

Use --pk-as-binary to expose __primary_key as binary with the direct reader. On Unix, --pprof-file <SVG> writes a flamegraph. Add --pprof-after-warmup and use at least two iterations to exclude the first iteration from profiling.

Replace an existing region SST

Warning: sst-replace mutates both an SST object and its region manifest. Stop the datanode that owns the region and back up the target SST and manifest before using --confirm. The SST write happens before the manifest update, so an interrupted or failed operation may require restoring the backup.

sst-replace replaces the contents of an existing SST file ID. It does not add a new file ID to a region. The command requires the replacement to have the same row count and row-group count recorded in the manifest when those manifest values are nonzero. It does not validate schema or row contents. The replacement is loaded into memory in full, so ensure the machine has enough memory for the SST.

Start with the default dry run using a local replacement file:

$GREPTIME datanode sst-replace \
  --config /path/to/datanode.toml \
  --region-id 1024:0 \
  --table-dir data/greptime/public/1024 \
  --file-id 00020380-009c-426d-953e-b4e34c15af34 \
  --replacement-file /tmp/rewritten.parquet

The dry run locates the manifest and target SST, reads and validates the replacement footer, and prints the old and new sizes without writing anything. --path-type defaults to auto, which probes bare, data, and metadata. Specify the path type if the file ID is visible in more than one manifest.

The replacement can instead be read from the configured object store:

$GREPTIME datanode sst-replace \
  --config /path/to/datanode.toml \
  --region-id 1024:0 \
  --table-dir data/greptime/public/1024 \
  --file-id 00020380-009c-426d-953e-b4e34c15af34 \
  --replacement-object staging/rewritten.parquet \
  --path-type bare

After reviewing the dry-run output and confirming that the datanode is stopped, repeat the exact command with --confirm:

$GREPTIME datanode sst-replace \
  --config /path/to/datanode.toml \
  --region-id 1024:0 \
  --table-dir data/greptime/public/1024 \
  --file-id 00020380-009c-426d-953e-b4e34c15af34 \
  --replacement-file /tmp/rewritten.parquet \
  --path-type bare \
  --confirm

The confirmed operation overwrites the existing SST object, recalculates file size and row-group statistics, and appends a manifest edit for the existing file ID. Restart the datanode and validate queries against the region before removing the backup.

Recommended rewrite and replacement workflow

1. Back up the source SST and its region manifest.
2. Inspect the source with parquet-meta.
3. Dump and edit writer properties with parquet-rewrite.
4. Rewrite to a new local file; never rewrite directly over the source SST.
5. Inspect the output and benchmark it with parquetbench.
6. Stop the owning datanode.
7. Run sst-replace without --confirm and review its resolved target and statistics.
8. Repeat with --confirm, restart the datanode, and validate the region.