mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-13 07:12:27 +00:00
fix: handle remote blob fetch request caps (#4309)
## Cause Remote `fetch_blobs` sent every row ID in one Cloud request, exceeding the server's 1024-row or 64 MiB byte cap. A single blob over the byte cap could not be read through this method. ## Fix - Send at most 1024 row IDs per request and split batches further when the server reports the blob-byte cap, preserving input order and nulls. - Resolve one exact dataset version before multi-request reads. If an initial single request hits the byte cap, resolve the version before any successful split or Range read; use that version for every resulting request. - Read an individual oversized blob through the existing Range route. - Document the per-request caps and transparent handling in the Rust, Python, and TypeScript APIs and generated JS docs. ## Validation - Focused Rust `fetch_blobs` tests: 26 passed, including row-count, byte-cap, Range fallback, and exact-version assertions. - Rust clippy and formatting passed. - Focused Python blob tests: 13 passed; Ruff lint and formatting passed. - Node build, lint, docs generation, and focused `fetchBlobs` test passed. Fixes #4275 <!-- lance-gatekeeper-fix:v1 agent=6ae94d49bf1d4cae057bba72584e7b90 generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
This commit is contained in:
@@ -537,6 +537,10 @@ export abstract class Table {
|
||||
* Reads the table's current checkout. IDs from another version can fail after
|
||||
* compaction unless stable row ids are enabled. Results keep input order and
|
||||
* duplicates. Null blobs are `null`. Empty blobs are empty buffers.
|
||||
* Remote servers limit each request to 1024 row IDs and 64 MiB of blob bytes.
|
||||
* The client splits requests automatically and reads an individual larger
|
||||
* blob through the Range route. This method still materializes all bytes in
|
||||
* memory; use {@link Table.fetchBlobFiles} for large values.
|
||||
*/
|
||||
abstract fetchBlobs(
|
||||
column: string,
|
||||
|
||||
Reference in New Issue
Block a user