mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-10-29 07:06:20 +00:00
dennis zhuang
5b4efd908c
* perf(mito2): build encoded bulk part indexes from in-memory bytes during flush Sync flush uploaded encoded bulk parts without indexes and rebuilt them after the flush, which read every encoded SST back from the object store and needed a second manifest commit. Build the indexes from the encoded bytes still in memory before the flush commits, the same way the regular SST writer does, and drop the post-flush rebuild for these files. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(mito2): keep flush going when indexing encoded bytes fails Match ParquetWriter: a failure while building indexes from the encoded bytes logs and leaves the SST unindexed instead of failing the flush after the SST was uploaded. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(mito2): own encoded SSTs for cleanup before uploading them Index creation now runs after an encoded SST is uploaded, so a truncate or drop could cancel the flush after the upload but before UncommittedSsts tracked the file, leaving the SST and its index behind. Track the file before the upload, and fill the index metadata on the caller's SstInfo instead of a partial one. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(mito2): name encoded SST index build params explicitly Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(mito2): drop index metadata when the puffin file fails and size index decode batches Indexer::do_finish kept the per-index metadata when finishing the puffin file failed, so callers could publish indexes that were not written. Return an empty output instead. Indexing encoded bulk parts now decodes with the batch size estimated from the largest row group, the same estimate bulk merges use, instead of a fixed 8192 rows. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs(config): document the mito index build_mode option Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(mito2): decode encoded SSTs for indexing on the blocking pool Indexing encoded bulk parts decoded Parquet batches inside the flush task on the global async runtime. Decode each batch with spawn_blocking_global and keep the indexer update on the async task, with one batch in flight. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com>