feat: add incremental primary key index writer (#8788)

* feat: initial implementation of the pk index writer

Signed-off-by: evenyag <realevenyag@gmail.com>

* perf: optimize primary key index writer

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: add todo

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat(mito2): track series count in pk index metrics

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor(mito2): simplify pk index writer cleanup

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix(mito2): clean up aborted pk index writers

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>

src/mito2/AGENTS.md

@@ -26,6 +26,7 @@ snapshot isolation). It implements the `RegionEngine` trait from `store-api`.
 | `compaction` | `src/mito2/src/compaction/` | Compaction scheduler (`scheduler.rs` + `scheduler/`), TWCS picker, strict-window manual picker, compactor, memory control |
 | `access_layer` | `src/mito2/src/access_layer.rs` | SST read/write over the object store |
 | `sst` | `src/mito2/src/sst/` | Parquet format, file metadata, index layout |
+| `pk_index` | `src/mito2/src/pk_index/` | Incremental writer for aggregate primary-key index files |
 | `read` | `src/mito2/src/read/` | `ScanRegion`, merge, dedup, projection, streaming |
 | `manifest` | `src/mito2/src/manifest/` | `RegionManifestManager`, manifest actions/edits |
 | `cache` | `src/mito2/src/cache.rs` | Write/file/page caches |

src/mito2/src/lib.rs

@@ -38,6 +38,7 @@ pub mod gc;
 pub mod manifest;
 pub mod memtable;
 mod metrics;
+pub mod pk_index;
 pub mod read;
 pub mod region;
 mod region_write_ctx;

src/mito2/src/pk_index.rs

@@ -0,0 +1,19 @@
+// Copyright 2023 Greptime Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Primary-key index writer.
+
+mod writer;
+
+pub use writer::{PkIndexWriter, PkIndexWriterMetrics, PkIndexWriterOptions, pk_columns_schema};