mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-16 15:46:55 +00:00
feat: support nullable scalar Function declarations (#5030)
Function declarations currently reject nullable scalar returns. Accept `Optional[T]`, `T | None`, and explicit nullable Arrow fields, preserving `nullable: true` in registration signatures, JSON round trips, and remote `add_columns` requests. The Rust planner accepts nullable scalar outputs, and binding readback recognizes the server-owned assignment marker for a single nullable scalar result. Clients do not allocate that marker. Non-nullable whole results, whole structs, and Blob v2 retain their existing assignment restrictions. Validated with 127 Python tests, 30 documentation tests (1 skipped), and seven focused Rust tests covering planning, binding readback, rejection boundaries, and the remote request body. The final condition-collapse change addresses strict Clippy; its delta check and CI are pending. Python tests use a source overlay on a retained native extension, and Rust remote coverage uses HTTP mocks. Server support and end-to-end validation remain separate. This PR does not claim that server publication distinguishes computed NULL values from pending rows.
This commit is contained in:
@@ -154,13 +154,27 @@ def test_function_output_nullability():
|
||||
("note", True),
|
||||
]
|
||||
|
||||
# A scalar output, and a struct declared as one nullable field, may not be.
|
||||
for nullable in (
|
||||
pa.field("result", pa.float64(), nullable=True),
|
||||
pa.field("result", pa.struct(fields), nullable=True),
|
||||
):
|
||||
with pytest.raises(ValueError, match="non-nullable"):
|
||||
function(input_schema=inputs, output_schema=nullable)(describe)
|
||||
# A scalar output may be nullable.
|
||||
@function(input_schema=inputs)
|
||||
def maybe_double(value: float) -> float | None:
|
||||
return None if value == 0 else value * 2
|
||||
|
||||
output = maybe_double.registration_request.signature.output
|
||||
assert (output.kind, output.arrow_type, output.nullable) == (
|
||||
"scalar",
|
||||
"float64",
|
||||
True,
|
||||
)
|
||||
assert json.loads(maybe_double.registration_request.signature.to_canonical_json())[
|
||||
"output"
|
||||
] == {"kind": "scalar", "arrow_type": "float64", "nullable": True}
|
||||
|
||||
# A struct declared as one nullable field may not be.
|
||||
with pytest.raises(ValueError, match="non-nullable"):
|
||||
function(
|
||||
input_schema=inputs,
|
||||
output_schema=pa.field("result", pa.struct(fields), nullable=True),
|
||||
)(describe)
|
||||
|
||||
|
||||
def test_function_registration_is_remote(tmp_path):
|
||||
|
||||
@@ -295,11 +295,12 @@ extract_frame_version = db.create_function_async(extract_frame).wait(
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
An empty byte string is a filled result. A scalar output cannot be nullable,
|
||||
because a physical NULL marks an unfilled computed row. To return a nullable
|
||||
blob as a value, use a named-struct output schema with a nullable blob field;
|
||||
the struct itself records whether the result has been filled. Read a nested
|
||||
blob with `table.fetch_blobs("payload.image", hits)`.
|
||||
An empty byte string is a filled result. A top-level Blob v2 output cannot be
|
||||
nullable; to return a nullable blob as a value, use a named-struct output
|
||||
schema with a nullable blob field. The struct itself records whether the
|
||||
result has been filled. A nullable non-Blob scalar output is declared with
|
||||
`Optional[T]` or `T | None`; see [Functions](/management/function#define).
|
||||
Read a nested blob with `table.fetch_blobs("payload.image", hits)`.
|
||||
|
||||
A list or large-list output may also contain Blob v2 elements; inputs and
|
||||
fixed-size lists do not support this layout. List-nested values are queryable
|
||||
|
||||
@@ -66,11 +66,43 @@ your process, without the database, so unit tests stay ordinary tests:
|
||||
|
||||
Where annotations cannot express the Arrow type precisely, pass `input_schema`
|
||||
and `output_schema` together: one without the other is rejected. A scalar
|
||||
output must be non-nullable, because a physical NULL is what marks a
|
||||
computed-column row as unfilled. A named-struct output may declare nullable
|
||||
fields: whether a row has a result at all is the struct's own validity, which
|
||||
Enterprise keeps in a separate internal column when it expands the result into
|
||||
sibling columns, so a null field is a value rather than a missing row.
|
||||
output is non-nullable by default. Annotate the return as `Optional[T]` or
|
||||
`T | None`, or pass a nullable output field, to declare a nullable scalar:
|
||||
|
||||
```python
|
||||
import pyarrow as pa
|
||||
|
||||
from lancedb import function
|
||||
|
||||
|
||||
@function(
|
||||
input_schema=pa.schema([pa.field("value", pa.int64(), nullable=False)]),
|
||||
output_schema=pa.field("result", pa.int64(), nullable=True),
|
||||
)
|
||||
def maybe_double(value: int) -> int | None:
|
||||
return None if value == 0 else value * 2
|
||||
```
|
||||
|
||||
The signature sent in the registration request then carries the nullable
|
||||
scalar:
|
||||
|
||||
```json
|
||||
{
|
||||
"inputs": [{"name": "value", "arrow_type": "int64", "nullable": false}],
|
||||
"output": {"kind": "scalar", "arrow_type": "int64", "nullable": true}
|
||||
}
|
||||
```
|
||||
|
||||
<Note>
|
||||
Nullable scalar outputs require an Enterprise server that supports them;
|
||||
older servers reject the registration. On a supporting server, a computed
|
||||
`NULL` counts as a filled result, independently of its stored value.
|
||||
</Note>
|
||||
|
||||
A named-struct output may declare nullable fields: whether a row has a result
|
||||
at all is the struct's own validity, which Enterprise keeps in a separate
|
||||
internal column when it expands the result into sibling columns, so a null
|
||||
field is a value rather than a missing row.
|
||||
|
||||
## Connect and create a table {#table}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user