fix(python): don't crash inferring schema for an all-null vector column (#4034)

Fixes #4033

`create_table(data=...)` without an explicit schema raised `IndexError:
index out of bounds` when a `vector`/`embedding`-named list column was
all-null. `_modal_list_size` infers the dimension with
`pc.mode(pc.list_value_length(arr))[0]`, and pyarrow's `mode` skips
nulls, so an all-null column produces a zero-length result and the `[0]`
subscript blows up.

The sibling `_infer_vector_dim` already guards the identical
`pc.mode(...)` with a `len == 0` check and returns `None`;
`_modal_list_size` just didn't. This makes `_modal_list_size` return
`Optional[int]` and return `None` when every value is null, and the call
site then leaves the column as a variable-length list instead of
inventing a fixed size — the dimension is genuinely unknowable when
there are no non-null rows.

Added a regression test that creates a table from an all-null
`list<float32>` vector column and asserts it succeeds, the column stays
a `list<float32>`, and the rows are queryable. It fails on `main` with
the `IndexError` and passes with this change.
This commit is contained in:
Madan Kumar
2026-10-23 01:10:45 +08:00
committed by GitHub
parent ae97d43f4b
commit 0c3a1c92c8
2 changed files with 38 additions and 3 deletions
+24
View File
@@ -280,6 +280,30 @@ def test_create_table_infers_large_int_vectors(mem_db: DBConnection):
assert vector_column.to_pylist() == [[0.0, 300.0]]
def test_create_table_all_null_vector_column(mem_db: DBConnection):
# A vector column can legitimately arrive with every value null, e.g. when
# the embeddings have not been computed yet. The dimension is unknowable in
# that case, so schema inference must not crash and should leave the column
# as a variable-length list rather than a fixed-size one.
data = pa.table(
{
"id": pa.array([1, 2]),
"vector": pa.array([None, None], type=pa.list_(pa.float32())),
}
)
table = mem_db.create_table("all_null_vector", data=data)
vector_field = table.schema.field("vector")
assert pa.types.is_list(vector_field.type)
assert vector_field.type.value_type == pa.float32()
result = table.to_arrow()
assert result.num_rows == 2
assert result.column("vector").to_pylist() == [None, None]
assert table.count_rows() == 2
@pytest.mark.asyncio
async def test_create_table_async_infers_large_int_vectors(
mem_db_async: AsyncConnection,