mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-12 23:02:28 +00:00
fix: handle remote blob fetch request caps (#4309)
## Cause Remote `fetch_blobs` sent every row ID in one Cloud request, exceeding the server's 1024-row or 64 MiB byte cap. A single blob over the byte cap could not be read through this method. ## Fix - Send at most 1024 row IDs per request and split batches further when the server reports the blob-byte cap, preserving input order and nulls. - Resolve one exact dataset version before multi-request reads. If an initial single request hits the byte cap, resolve the version before any successful split or Range read; use that version for every resulting request. - Read an individual oversized blob through the existing Range route. - Document the per-request caps and transparent handling in the Rust, Python, and TypeScript APIs and generated JS docs. ## Validation - Focused Rust `fetch_blobs` tests: 26 passed, including row-count, byte-cap, Range fallback, and exact-version assertions. - Rust clippy and formatting passed. - Focused Python blob tests: 13 passed; Ruff lint and formatting passed. - Node build, lint, docs generation, and focused `fetchBlobs` test passed. Fixes #4275 <!-- lance-gatekeeper-fix:v1 agent=6ae94d49bf1d4cae057bba72584e7b90 generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
This commit is contained in:
@@ -1963,8 +1963,10 @@ class Table(ABC):
|
||||
``_rowid`` values stay valid after compaction when the table has stable
|
||||
row ids.
|
||||
|
||||
Convenience for small payloads. For large values use
|
||||
:meth:`fetch_blob_files`.
|
||||
Remote servers limit each request to 1024 row IDs and 64 MiB of blob
|
||||
bytes. The client splits requests automatically and reads an individual
|
||||
larger blob through the Range route. This method still materializes all
|
||||
bytes in memory; for large values use :meth:`fetch_blob_files`.
|
||||
"""
|
||||
|
||||
@abstractmethod
|
||||
|
||||
Reference in New Issue
Block a user