feat: support nullable named function outputs (#4123)

Supports fully nullable named Function outputs while preserving the
distinction between a valid all-null struct and a null/unassigned
result.

## Concrete example

This UDF contract is now valid:

```python
@udf(
    input_schema=pa.schema([
        pa.field("text", pa.string(), nullable=False),
    ]),
    output_schema=pa.schema([
        pa.field(
            "embedding",
            pa.list_(pa.float32(), list_size=1024),
            nullable=True,
        ),
        pa.field("embedding_failure_reason", pa.string(), nullable=True),
        pa.field("embedding_failure_code", pa.int32(), nullable=True),
    ]),
)
def embed(text):
    ...
```

A successful row can return:

```text
embedding = [0.12, ...]
embedding_failure_reason = NULL
embedding_failure_code = NULL
```

If remote inference still fails after retries, it can return:

```text
embedding = NULL
embedding_failure_reason = "HTTP 429: rate limited"
embedding_failure_code = 429
```

An all-null but valid result struct is also assigned; it is not mistaken
for unfinished work.

## Binding shapes

- Mapping the result to one output column stores the `StructArray`
directly, including its parent validity bitmap.
- Flattening the result into top-level columns stores the parent
validity in a reserved internal nullable Boolean assignment column that
is not part of the UDF result mapping.
- An outer null struct remains unassigned/skipped. A valid struct
remains assigned regardless of which child fields are null.
- Scalar Function outputs remain non-nullable.

The contract is preserved through Python registration, Rust application
planning, persisted `FunctionBinding` metadata, schema revalidation, and
Enterprise execution.
This commit is contained in:
Jack Ye
2026-09-03 22:23:53 -07:00
committed by GitHub
parent e639b1b650
commit aab23eb39e
6 changed files with 270 additions and 33 deletions
+2
View File
@@ -147,6 +147,8 @@ listing a storage directory.
::: lancedb.functions.OutputMapping
::: lancedb.functions.AssignmentMapping
::: lancedb.functions.FunctionBinding
::: lancedb.functions.RefreshColumnResult