rust/lancedb
1
0
Fork 0
mirror of https://github.com/lancedb/lancedb.git synced 2026-11-15 07:06:52 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(python): support native blob conversion in pandas and batches (#4647)

Browse Source 
Plain local blob v2 scans tried to open a pylance dataset before
reaching LanceDB's byte-fetch path, so `to_pandas(blob_mode="bytes")`
failed when the optional dependency was absent. Batch readers also had
no `blob_mode` option.

Route supported blob v2 pandas conversions through the native Arrow
path. Add `blob_mode` to sync and async batch readers, convert payloads
during iteration, and expose the converted schema without fetching the
first batch. Apply conversion after reranking and retain the default
descriptor behavior. Clean up scanner error punctuation and remove the
misleading plain-scan suggestion.

Bytes mode retains the existing `to_arrow` restrictions for legacy/list
blobs, remote tables, and namespace query pushdown. Lazy pandas handles
and legacy pandas scans still require pylance; sync FTS/hybrid batching
remains unsupported.

Validation:
- UV-managed editable Python development build completed successfully.
- The absent-pylance regression tests fail against the original query
implementation and pass with this change.
- Blob and query suites: 403 passed, 1 skipped, 1 deselected.
- Remote query suite: 190 passed.
- Python-tree Ruff lint/format checks and the async reader Pyright check
pass. Repository-wide Ruff reports existing errors outside the Python
tree.

Fixes #4645

<!-- lance-gatekeeper-fix:v1 agent=fd85998f14bb0b3c92b2d050df93a615
generation=1 -->

---------

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
This commit is contained in:
lancedb-gatefixer[bot]
2026-10-28 02:59:02 +08:00
committed by GitHub
co-authored by Xuanwo
parent 6e4f523e14
commit ac99ab173b
7 changed files with 419 additions and 43 deletions
Download Patch File Download Diff File Expand all files Collapse all files
python/python/tests/test_query.py
+3 -3
View File
@@ -675,10 +675,10 @@ def test_blob_v2_vector_projection_and_plain_scan_options(tmp_db, monkeypatch):
query = table.search().where("id = 1")
def fail_to_arrow(*args, **kwargs):
raise AssertionError("the native scanner should handle bytes with flatten")
def fail_to_lance(*args, **kwargs):
raise AssertionError("blob v2 bytes should not open a pylance dataset")
monkeypatch.setattr(query, "to_arrow", fail_to_arrow)
monkeypatch.setattr(table, "to_lance", fail_to_lance)
assert query.to_pandas(blob_mode="bytes", flatten=True)["blob"].tolist() == [b"one"]
flattened_lazy = table.search().where("id = 1").to_pandas(flatten=True)