fix(python): add a Blob V2 array construction helper (#4734)

Fixes #4733

PyArrow expects Blob V2 storage structs when constructing a record batch
and cannot infer the extension array from raw bytes. LanceDB already
converts binary values during ingestion, but batch writers had no
supported array constructor.

Add `lancedb.schema.blob_array(payloads)` for iterables of inline bytes
and nulls, using the same extension type resolver as `blob()`. The
helper preserves empty blobs and null blobs and works with both
pylance's type and LanceDB's fallback. Document the descriptor fields,
exclusive inline/external representations, batch construction, and
existing binary conversion in synchronous and asynchronous
`create_table` / `add` APIs.

Validation:
- Reproduced the reported `ArrowTypeError` on the original code; both
new regression cases fail against the original schema module.
- 256 focused blob tests and schema doctests passed; one optional PyAV
test skipped.
- Both new regression cases also passed with PyArrow 25, with and
without pylance imports.
- Python package Ruff checks and formatting passed.
- MkDocs build passed, with an existing unrelated `SecretReference`
cross-reference warning. Repository-wide Ruff reports 48 existing errors
in `docs/web-tests`.

<!-- lance-gatekeeper-fix:v1 agent=20d1a296d76ff5eb914c0848428a747a
generation=1 -->

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
This commit is contained in:
lancedb-gatefixer[bot]
2026-10-29 11:28:54 -07:00
committed by GitHub
parent dcddd541c8
commit a4f52cd205
5 changed files with 198 additions and 1 deletions
+30
View File
@@ -589,6 +589,34 @@ with `search().with_row_id(True).to_arrow()` and pass the result to the fetch AP
One missing or deleted row ID rejects the entire blob-read batch; the APIs do
not return partial results.
To construct inline Blob V2 arrays for ingestion in record batches, use
[blob_array][lancedb.schema.blob_array]. It accepts bytes and `None`, preserving
the distinction between an empty blob (`b""`) and a null blob:
```python
import pyarrow as pa
from lancedb.schema import blob, blob_array
schema = pa.schema([pa.field("video_id", pa.string()), blob("video")])
batch = pa.RecordBatch.from_arrays(
[pa.array(["a", "b", "c"]), blob_array([b"abc", None, b""])],
schema=schema,
)
table = db.create_table("videos", data=iter([batch]), schema=schema)
table.add(iter([batch]))
```
`create_table(..., schema=...)` also casts plain binary values into blob columns
declared in the schema. `table.add(...)` casts them using the table's schema.
These conversions apply to Python rows and plain binary Arrow columns.
PyArrow's `RecordBatch.from_pylist(..., schema=...)` cannot convert raw bytes
into the blob extension type; use `blob_array` and `RecordBatch.from_arrays`
when constructing batches explicitly.
For external blob references, construct the descriptor struct described by
[blob][lancedb.schema.blob], then wrap it with `pa.ExtensionArray.from_storage`.
Inline `data` and external `uri` representations are mutually exclusive.
LanceDB Distributed SQL returns blob bytes by default. Select `blob_descriptor(video)` to
return an openable reference instead, then use
[`open_blob_file`][lancedb.table.Table.open_blob_file] or the lazy iterable
@@ -647,6 +675,8 @@ so the fetch APIs cannot read their bytes yet.
::: lancedb.blob
::: lancedb.schema.blob_array
::: lancedb._blob.BlobFile
options:
show_root_full_path: false