fix: share scans across batched vector queries (#3805)

<!-- lance-gatekeeper-fix:v1 agent=d30696bc46eb32f04c9927b0792e35d3
generation=1 -->

## Summary

- use the Lance native batch KNN path so fixed-size batch vector
searches share one flat table scan
- validate consistent query-vector dimensions and retain the per-vector
plan when offsets require its existing semantics
- add Rust and Python regressions and update Rust, Python, and
TypeScript API documentation

## Root cause

LanceDB expanded every vector in a batch into a separate scan plan and
joined the plans with `UnionExec`. For unindexed tables on S3, a batch
of ten vectors therefore ran ten concurrent full scans, amplifying CPU
and retained data enough to produce the reported memory spike.

The native Lance batch KNN path performs bounded-memory selection for
all query vectors over one flat scan. LanceDB now supplies the vectors
as a batch and avoids applying a global scanner limit to the combined
per-query results. Batch queries with a nonzero offset keep the previous
plan because the native batch API does not support per-query offsets.

## Validation

- targeted Rust batch-query plan and execution tests
- `cargo check --quiet --features remote --tests --examples`
- `cargo clippy --quiet --features remote --tests --examples`
- `cargo fmt --all -- --check`
- targeted Python batch-vector regression after rebuilding the extension
- Ruff formatting/checks for the touched Python files
- Node.js build, lint, docs generation, and targeted batch-vector Jest
test
- `git diff --check`

Fixes #2468

---------

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>

python/python/lancedb/query.py

@@ -3401,9 +3401,10 @@ class AsyncQuery(AsyncStandardQuery):
         pass in multiple vectors. When multiple vectors are passed in, if the vector
         column is with multivector type, then the vectors will be treated as a single
         query. Or the vectors will be treated as multiple queries, this can be useful
-        if you want to find the nearest vectors to multiple query vectors.
-        This is not expected to be faster than making multiple queries concurrently;
-        it is just a convenience method. If multiple vectors are passed in then
+        if you want to find the nearest vectors to multiple query vectors. Flat
+        searches share one table scan across the query vectors, avoiding the scan
+        and memory amplification of making multiple queries concurrently. If
+        multiple vectors are passed in then
         an additional column `query_index` will be added to the results. This column
         will contain the index of the query vector that the result is nearest to.
         """
@@ -3532,8 +3533,8 @@ class AsyncFTSQuery(AsyncStandardQuery):
 
         Typically, a single vector is passed in as the query. However, you can also
         pass in multiple vectors.  This can be useful if you want to find the nearest
-        vectors to multiple query vectors. This is not expected to be faster than
-        making multiple queries concurrently; it is just a convenience method.
+        vectors to multiple query vectors. Flat searches share one table scan across
+        the query vectors instead of issuing concurrent full scans.
         If multiple vectors are passed in then an additional column `query_index`
         will be added to the results.  This column will contain the index of the
         query vector that the result is nearest to.

python/python/tests/test_query.py

@@ -897,6 +897,23 @@ def test_query_builder_batches(table):
     assert rs_list["id"][1] == 2
 
 
+def test_batch_vector_query_shares_filtered_flat_scan(table):
+    query = (
+        table.search([[1.0, 2.0], [3.0, 4.0]])
+        .where("id > 0", prefilter=True)
+        .limit(1)
+        .select(["id"])
+    )
+
+    plan = query.explain_plan(verbose=True)
+    assert "KNNVectorDistance: queries=2" in plan
+    assert "UnionExec" not in plan
+
+    results = query.to_arrow()
+    assert len(results) == 2
+    assert results["query_index"].to_pylist() == [0, 1]
+
+
 def test_dynamic_projection(table):
     rs = (
         LanceVectorQueryBuilder(table, [0, 0], "vector")