mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-29 06:56:15 +00:00
fix(python): support native blob conversion in pandas and batches (#4647)
Plain local blob v2 scans tried to open a pylance dataset before reaching LanceDB's byte-fetch path, so `to_pandas(blob_mode="bytes")` failed when the optional dependency was absent. Batch readers also had no `blob_mode` option. Route supported blob v2 pandas conversions through the native Arrow path. Add `blob_mode` to sync and async batch readers, convert payloads during iteration, and expose the converted schema without fetching the first batch. Apply conversion after reranking and retain the default descriptor behavior. Clean up scanner error punctuation and remove the misleading plain-scan suggestion. Bytes mode retains the existing `to_arrow` restrictions for legacy/list blobs, remote tables, and namespace query pushdown. Lazy pandas handles and legacy pandas scans still require pylance; sync FTS/hybrid batching remains unsupported. Validation: - UV-managed editable Python development build completed successfully. - The absent-pylance regression tests fail against the original query implementation and pass with this change. - Blob and query suites: 403 passed, 1 skipped, 1 deselected. - Remote query suite: 190 passed. - Python-tree Ruff lint/format checks and the async reader Pyright check pass. Repository-wide Ruff reports existing errors outside the Python tree. Fixes #4645 <!-- lance-gatekeeper-fix:v1 agent=fd85998f14bb0b3c92b2d050df93a615 generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io>
This commit is contained in:
co-authored by
Xuanwo
parent
6e4f523e14
commit
ac99ab173b
@@ -447,8 +447,8 @@ above:
|
||||
</CodeGroup>
|
||||
|
||||
In the Python SDK, `Table.to_pandas()`, `Table.to_arrow()`, and the query builder
|
||||
methods `to_pandas()`, `to_arrow()`, and `to_list()` accept a `blob_mode`
|
||||
argument:
|
||||
methods `to_pandas()`, `to_arrow()`, `to_list()`, and `to_batches()` accept a
|
||||
`blob_mode` argument. Sync and async query builders support the same modes:
|
||||
|
||||
| `blob_mode` | `to_pandas()` | `to_arrow()` and `to_list()` |
|
||||
| --- | --- | --- |
|
||||
@@ -456,10 +456,11 @@ argument:
|
||||
| `"descriptions"` | Returns descriptor dicts with `kind`, `position`, `size`, `blob_id`, and `blob_uri`. | Returns the 5-field descriptor without `_lance_row_id`. |
|
||||
| `"bytes"` | Materializes each blob as `bytes` in the DataFrame. | Materializes each blob as `large_binary` in Arrow or `bytes` in Python dicts. |
|
||||
|
||||
`to_pandas(blob_mode="lazy")` and `to_pandas(blob_mode="bytes")` require
|
||||
pylance. Without it they raise `ImportError`. `to_arrow()`,
|
||||
`to_list(blob_mode="bytes")`, `to_pandas(blob_mode="descriptions")`, and the
|
||||
fetch methods do not.
|
||||
For supported local v2 blob projections, `to_pandas(blob_mode="bytes")` and
|
||||
`to_pandas(blob_mode="descriptions")` use native Arrow conversion without
|
||||
pylance. Pandas conversions that need the Lance scanner, including plain-scan
|
||||
`"lazy"` handles and legacy blobs, still require pylance. `to_arrow()`,
|
||||
`to_list()`, `to_batches()`, and the fetch methods do not require pylance.
|
||||
|
||||
`blob_mode="bytes"` requires a local table snapshot, so remote and in-memory
|
||||
tables raise `NotImplementedError`. On a remote table, `Table.to_pandas()` and
|
||||
@@ -475,9 +476,34 @@ them out of the projection. Extra keyword arguments on
|
||||
`to_pandas()`, such as `split_blocks` and `self_destruct`, are forwarded to
|
||||
PyArrow.
|
||||
|
||||
###### Stream blob results {#multimodal-blob-batches}
|
||||
|
||||
`to_batches(blob_mode="bytes")` fetches blob v2 payloads one batch at a time
|
||||
and returns them as Arrow `large_binary` values. Iterate over the reader and
|
||||
call `batch.to_pandas()` to convert each batch to a DataFrame without collecting
|
||||
all payloads at once.
|
||||
|
||||
For sync queries, call `query.to_batches(batch_size=64, blob_mode="bytes")`
|
||||
and iterate with `for`. For async queries, call
|
||||
`await query.to_batches(max_batch_length=64, blob_mode="bytes")` and iterate
|
||||
with `async for`. Sync full-text and hybrid queries do not support
|
||||
`to_batches()`.
|
||||
|
||||
The default `"lazy"` mode returns descriptors and keeps automatic `_rowid`
|
||||
columns visible for manual fetches. `"descriptions"` returns the five-field
|
||||
descriptors. Both `"bytes"` and `"descriptions"` hide automatic row IDs;
|
||||
explicitly requested row IDs remain visible.
|
||||
|
||||
Bytes mode has the same restrictions as `to_arrow(blob_mode="bytes")`,
|
||||
including no legacy blobs, blobs inside lists, remote tables, or namespace
|
||||
`QueryTable` pushdown. If concurrent writers can overwrite the table, use a
|
||||
separate reader checked out to a fixed version before querying. Otherwise,
|
||||
an overwrite can invalidate the row IDs used to fetch bytes and raise
|
||||
`ValueError`.
|
||||
|
||||
###### Optional pandas examples {#multimodal-blob-pandas}
|
||||
|
||||
These examples require pylance. The query example expects `tbl_async`,
|
||||
The lazy-handle calls below require pylance. The query example expects `tbl_async`,
|
||||
`query_vector`, and a `video` column, which the `photos` table does not define.
|
||||
|
||||
<CodeGroup>
|
||||
|
||||
Reference in New Issue
Block a user