mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-13 14:16:54 +00:00
fix(python): add a Blob V2 array construction helper (#4734)
Fixes #4733 PyArrow expects Blob V2 storage structs when constructing a record batch and cannot infer the extension array from raw bytes. LanceDB already converts binary values during ingestion, but batch writers had no supported array constructor. Add `lancedb.schema.blob_array(payloads)` for iterables of inline bytes and nulls, using the same extension type resolver as `blob()`. The helper preserves empty blobs and null blobs and works with both pylance's type and LanceDB's fallback. Document the descriptor fields, exclusive inline/external representations, batch construction, and existing binary conversion in synchronous and asynchronous `create_table` / `add` APIs. Validation: - Reproduced the reported `ArrowTypeError` on the original code; both new regression cases fail against the original schema module. - 256 focused blob tests and schema doctests passed; one optional PyAV test skipped. - Both new regression cases also passed with PyArrow 25, with and without pylance imports. - Python package Ruff checks and formatting passed. - MkDocs build passed, with an existing unrelated `SecretReference` cross-reference warning. Repository-wide Ruff reports 48 existing errors in `docs/web-tests`. <!-- lance-gatekeeper-fix:v1 agent=20d1a296d76ff5eb914c0848428a747a generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
This commit is contained in:
@@ -589,6 +589,34 @@ with `search().with_row_id(True).to_arrow()` and pass the result to the fetch AP
|
||||
One missing or deleted row ID rejects the entire blob-read batch; the APIs do
|
||||
not return partial results.
|
||||
|
||||
To construct inline Blob V2 arrays for ingestion in record batches, use
|
||||
[blob_array][lancedb.schema.blob_array]. It accepts bytes and `None`, preserving
|
||||
the distinction between an empty blob (`b""`) and a null blob:
|
||||
|
||||
```python
|
||||
import pyarrow as pa
|
||||
from lancedb.schema import blob, blob_array
|
||||
|
||||
schema = pa.schema([pa.field("video_id", pa.string()), blob("video")])
|
||||
batch = pa.RecordBatch.from_arrays(
|
||||
[pa.array(["a", "b", "c"]), blob_array([b"abc", None, b""])],
|
||||
schema=schema,
|
||||
)
|
||||
table = db.create_table("videos", data=iter([batch]), schema=schema)
|
||||
table.add(iter([batch]))
|
||||
```
|
||||
|
||||
`create_table(..., schema=...)` also casts plain binary values into blob columns
|
||||
declared in the schema. `table.add(...)` casts them using the table's schema.
|
||||
These conversions apply to Python rows and plain binary Arrow columns.
|
||||
PyArrow's `RecordBatch.from_pylist(..., schema=...)` cannot convert raw bytes
|
||||
into the blob extension type; use `blob_array` and `RecordBatch.from_arrays`
|
||||
when constructing batches explicitly.
|
||||
|
||||
For external blob references, construct the descriptor struct described by
|
||||
[blob][lancedb.schema.blob], then wrap it with `pa.ExtensionArray.from_storage`.
|
||||
Inline `data` and external `uri` representations are mutually exclusive.
|
||||
|
||||
LanceDB Distributed SQL returns blob bytes by default. Select `blob_descriptor(video)` to
|
||||
return an openable reference instead, then use
|
||||
[`open_blob_file`][lancedb.table.Table.open_blob_file] or the lazy iterable
|
||||
@@ -647,6 +675,8 @@ so the fetch APIs cannot read their bytes yet.
|
||||
|
||||
::: lancedb.blob
|
||||
|
||||
::: lancedb.schema.blob_array
|
||||
|
||||
::: lancedb._blob.BlobFile
|
||||
options:
|
||||
show_root_full_path: false
|
||||
|
||||
Reference in New Issue
Block a user