mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-10-29 15:06:59 +00:00
dennis zhuang
d18382a33d
perf(mito2): index encoded bulk parts from memory during flush (#9913)
* perf(mito2): build encoded bulk part indexes from in-memory bytes during flush
Sync flush uploaded encoded bulk parts without indexes and rebuilt them after
the flush, which read every encoded SST back from the object store and needed a
second manifest commit. Build the indexes from the encoded bytes still in memory
before the flush commits, the same way the regular SST writer does, and drop the
post-flush rebuild for these files.
* fix(mito2): keep flush going when indexing encoded bytes fails
Match ParquetWriter: a failure while building indexes from the encoded bytes
logs and leaves the SST unindexed instead of failing the flush after the SST
was uploaded.
* fix(mito2): own encoded SSTs for cleanup before uploading them
Index creation now runs after an encoded SST is uploaded, so a truncate or drop
could cancel the flush after the upload but before UncommittedSsts tracked the
file, leaving the SST and its index behind. Track the file before the upload,
and fill the index metadata on the caller's SstInfo instead of a partial one.
* refactor(mito2): name encoded SST index build params explicitly
* fix(mito2): drop index metadata when the puffin file fails and size index decode batches
Indexer::do_finish kept the per-index metadata when finishing the puffin file
failed, so callers could publish indexes that were not written. Return an
empty output instead.
Indexing encoded bulk parts now decodes with the batch size estimated from the
largest row group, the same estimate bulk merges use, instead of a fixed 8192
rows.
* docs(config): document the mito index build_mode option
* fix(mito2): decode encoded SSTs for indexing on the blocking pool
Indexing encoded bulk parts decoded Parquet batches inside the flush task on
the global async runtime. Decode each batch with spawn_blocking_global and keep
the indexer update on the async task, with one batch in flight.
---------
(cherry picked from commit 5b4efd908c)
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>