fix(python): support native blob conversion in pandas and batches (#4647)

Plain local blob v2 scans tried to open a pylance dataset before
reaching LanceDB's byte-fetch path, so `to_pandas(blob_mode="bytes")`
failed when the optional dependency was absent. Batch readers also had
no `blob_mode` option.

Route supported blob v2 pandas conversions through the native Arrow
path. Add `blob_mode` to sync and async batch readers, convert payloads
during iteration, and expose the converted schema without fetching the
first batch. Apply conversion after reranking and retain the default
descriptor behavior. Clean up scanner error punctuation and remove the
misleading plain-scan suggestion.

Bytes mode retains the existing `to_arrow` restrictions for legacy/list
blobs, remote tables, and namespace query pushdown. Lazy pandas handles
and legacy pandas scans still require pylance; sync FTS/hybrid batching
remains unsupported.

Validation:
- UV-managed editable Python development build completed successfully.
- The absent-pylance regression tests fail against the original query
implementation and pass with this change.
- Blob and query suites: 403 passed, 1 skipped, 1 deselected.
- Remote query suite: 190 passed.
- Python-tree Ruff lint/format checks and the async reader Pyright check
pass. Repository-wide Ruff reports existing errors outside the Python
tree.

Fixes #4645

<!-- lance-gatekeeper-fix:v1 agent=fd85998f14bb0b3c92b2d050df93a615
generation=1 -->

---------

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
This commit is contained in:
lancedb-gatefixer[bot]
2026-10-28 02:59:02 +08:00
committed by GitHub
co-authored by Xuanwo
parent 6e4f523e14
commit ac99ab173b
7 changed files with 419 additions and 43 deletions
+33 -7
View File
@@ -447,8 +447,8 @@ above:
</CodeGroup>
In the Python SDK, `Table.to_pandas()`, `Table.to_arrow()`, and the query builder
methods `to_pandas()`, `to_arrow()`, and `to_list()` accept a `blob_mode`
argument:
methods `to_pandas()`, `to_arrow()`, `to_list()`, and `to_batches()` accept a
`blob_mode` argument. Sync and async query builders support the same modes:
| `blob_mode` | `to_pandas()` | `to_arrow()` and `to_list()` |
| --- | --- | --- |
@@ -456,10 +456,11 @@ argument:
| `"descriptions"` | Returns descriptor dicts with `kind`, `position`, `size`, `blob_id`, and `blob_uri`. | Returns the 5-field descriptor without `_lance_row_id`. |
| `"bytes"` | Materializes each blob as `bytes` in the DataFrame. | Materializes each blob as `large_binary` in Arrow or `bytes` in Python dicts. |
`to_pandas(blob_mode="lazy")` and `to_pandas(blob_mode="bytes")` require
pylance. Without it they raise `ImportError`. `to_arrow()`,
`to_list(blob_mode="bytes")`, `to_pandas(blob_mode="descriptions")`, and the
fetch methods do not.
For supported local v2 blob projections, `to_pandas(blob_mode="bytes")` and
`to_pandas(blob_mode="descriptions")` use native Arrow conversion without
pylance. Pandas conversions that need the Lance scanner, including plain-scan
`"lazy"` handles and legacy blobs, still require pylance. `to_arrow()`,
`to_list()`, `to_batches()`, and the fetch methods do not require pylance.
`blob_mode="bytes"` requires a local table snapshot, so remote and in-memory
tables raise `NotImplementedError`. On a remote table, `Table.to_pandas()` and
@@ -475,9 +476,34 @@ them out of the projection. Extra keyword arguments on
`to_pandas()`, such as `split_blocks` and `self_destruct`, are forwarded to
PyArrow.
###### Stream blob results {#multimodal-blob-batches}
`to_batches(blob_mode="bytes")` fetches blob v2 payloads one batch at a time
and returns them as Arrow `large_binary` values. Iterate over the reader and
call `batch.to_pandas()` to convert each batch to a DataFrame without collecting
all payloads at once.
For sync queries, call `query.to_batches(batch_size=64, blob_mode="bytes")`
and iterate with `for`. For async queries, call
`await query.to_batches(max_batch_length=64, blob_mode="bytes")` and iterate
with `async for`. Sync full-text and hybrid queries do not support
`to_batches()`.
The default `"lazy"` mode returns descriptors and keeps automatic `_rowid`
columns visible for manual fetches. `"descriptions"` returns the five-field
descriptors. Both `"bytes"` and `"descriptions"` hide automatic row IDs;
explicitly requested row IDs remain visible.
Bytes mode has the same restrictions as `to_arrow(blob_mode="bytes")`,
including no legacy blobs, blobs inside lists, remote tables, or namespace
`QueryTable` pushdown. If concurrent writers can overwrite the table, use a
separate reader checked out to a fixed version before querying. Otherwise,
an overwrite can invalidate the row IDs used to fetch bytes and raise
`ValueError`.
###### Optional pandas examples {#multimodal-blob-pandas}
These examples require pylance. The query example expects `tbl_async`,
The lazy-handle calls below require pylance. The query example expects `tbl_async`,
`query_vector`, and a `video` column, which the `photos` table does not define.
<CodeGroup>