mirror of
https://github.com/lancedb/lancedb.git
synced 2026-08-30 18:08:24 +00:00
lancedb-gatefixer[bot]
ae81d73563
<!-- lance-gatekeeper-fix:v1 agent=d30696bc46eb32f04c9927b0792e35d3 generation=1 --> ## Summary - use the Lance native batch KNN path so fixed-size batch vector searches share one flat table scan - validate consistent query-vector dimensions and retain the per-vector plan when offsets require its existing semantics - add Rust and Python regressions and update Rust, Python, and TypeScript API documentation ## Root cause LanceDB expanded every vector in a batch into a separate scan plan and joined the plans with `UnionExec`. For unindexed tables on S3, a batch of ten vectors therefore ran ten concurrent full scans, amplifying CPU and retained data enough to produce the reported memory spike. The native Lance batch KNN path performs bounded-memory selection for all query vectors over one flat scan. LanceDB now supplies the vectors as a batch and avoids applying a global scanner limit to the combined per-query results. Batch queries with a nonzero offset keep the previous plan because the native batch API does not support per-query offsets. ## Validation - targeted Rust batch-query plan and execution tests - `cargo check --quiet --features remote --tests --examples` - `cargo clippy --quiet --features remote --tests --examples` - `cargo fmt --all -- --check` - targeted Python batch-vector regression after rebuilding the extension - Ruff formatting/checks for the touched Python files - Node.js build, lint, docs generation, and targeted batch-vector Jest test - `git diff --check` Fixes #2468 --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io>