Will JonesandClaude Opus 5 1d2a5d084b fix: accept all-null batches and plain JSON strings for json columns (#4067) ...

Two ways of writing to a `json` column failed or silently corrupted
data.

**All-null batches were rejected.** `add()` refused a batch whose values
for a `json` column were all null, while every plain Arrow type accepted
the same batch. This bites row-at-a-time inserts hardest: a one-row
batch with no value for an optional column is trivially all-null, so
most such writes failed. pyarrow infers `null` as the column's type, and
the write path had no handling for it — casting to the table's type
dropped the field metadata that identifies the column as `lance.json`,
so lance rejected the batch (`` `val` should have type json but type was
large_binary ``). A null-typed input column now becomes typed nulls
matching the table's field exactly, metadata included.

**Unlabelled JSON text was stored raw.** JSON supplied as plain strings
(what pyarrow infers for a column of `str`) was cast to the column's
`LargeBinary` storage type and relabelled `lance.json`, putting unparsed
text where JSONB was expected. Reads returned the text unnormalized and
`json_extract` failed with `InvalidJsonb`. Lance-core does the JSONB
encoding, but only for input labelled `arrow.json`, so string input is
now labelled rather than cast — at the top level and inside structs.

Both fixes are in the shared Rust write path, so they apply to any
binding, including hand-built Arrow tables that never pass through
Python's list-of-dicts type inference. `_align_field` gets the same
JSON-string fix for the legacy Python `_sanitize_data` path, which
`on_bad_vectors` and embedding functions still route through.

The blob v2 half of the issue landed separately in #4065, which added a
`DataType::Null` arm to blob coercion. This PR keeps that implementation
and adds end-to-end add-path coverage for it.

The tests from #4066 are included here and pass, so that PR's
Python-layer inference changes are no longer needed to close the issue.

Fixes #3759

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

2026-09-15 15:46:32 -07:00

..

lancedb

fix: accept all-null batches and plain JSON strings for json columns (#4067)

2026-09-15 15:46:32 -07:00

license_header.txt

ci: check license headers (#2076)

2025-01-29 08:27:07 -08:00