rust/lancedb
1
0
Fork 0
mirror of https://github.com/lancedb/lancedb.git synced 2026-10-31 07:46:51 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(node): expose targetPartitionSize on IVF index options (#4594)

Browse Source 
The Node SDK's vector index options only take `numPartitions`. The Rust index builders (`target_partition_size` in `rust/lancedb/src/index/vector.rs`) and the Python SDK already support a target partition size, so this adds `targetPartitionSize` to `IvfPqOptions`, `IvfRqOptions`, `IvfFlatOptions`, `HnswPqOptions` and `HnswSqOptions` and passes it to their builders. When both are set, `numPartitions` is used, as it is for Rust and Python callers.

The reason to want it: an index built from a partition count records no target partition size, so `table.optimize()` splits and joins it against the index type's default and can merge small partitions down to one or two. Building with a target records it in the index details, and optimize keeps to it. https://github.com/lance-format/lance/pull/10154 fixes this on the Lance side for indexes built from a partition count; this lets Node users set the size directly.

The new tests build IvfPq, IvfFlat, IvfRq, IvfHnswPq and IvfHnswSq indexes with `targetPartitionSize` and check the size recorded in the index details, and check that `numPartitions` wins when both are set. Each option's doc comment states that index type's default target. Docs regenerated with `pnpm run docs`.
This commit is contained in:
geoffgscott
2026-10-29 13:13:00 +08:00
committed by GitHub
parent 03bd909dab
commit 7ea10fa1e0
10 changed files with 271 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files
nodejs/__test__/table.test.ts
+32
View File
@@ -1707,6 +1707,38 @@ describe("When creating an index", () => {
});
});
it.each([
["IvfPq", Index.ivfPq({ targetPartitionSize: 100 })],
["IvfFlat", Index.ivfFlat({ targetPartitionSize: 100 })],
["IvfRq", Index.ivfRq({ targetPartitionSize: 100 })],
["IvfHnswPq", Index.hnswPq({ targetPartitionSize: 100 })],
["IvfHnswSq", Index.hnswSq({ targetPartitionSize: 100 })],
])("should create %s with targetPartitionSize", async (indexType, config) => {
await tbl.createIndex("vec", { config });
const indices = await tbl.listIndices();
expect(indices).toHaveLength(1);
expect(indices[0].indexType).toBe(indexType);
expect(
(indices[0].indexDetails as Record<string, unknown>)[
"target_partition_size"
],
).toBe(100);
});
it("should prefer numPartitions over targetPartitionSize", async () => {
await tbl.createIndex("vec", {
config: Index.ivfFlat({ numPartitions: 2, targetPartitionSize: 100 }),
});
const indices = await tbl.listIndices();
expect(
(indices[0].indexDetails as Record<string, unknown>)[
"target_partition_size"
],
).toBeUndefined();
});
it("should allow me to replace (or not) an existing index", async () => {
await tbl.createIndex("id");
// Default is replace=true