mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-13 14:16:54 +00:00
docs: show how to run Functions and BlobReader locally (#4803)
Running a Function body in a local test or another engine currently requires discovering undocumented APIs. Document direct calls to decorated Functions and classes, including initialization fields, instance reuse, and cleanup, and explain how they differ from registered `FunctionVersion` applications. Add examples of `BlobReader(read_range, size)` backed by an in-memory byte-range callback, including cursor-independent reads. The local examples and rendered API documentation were validated on EC2 at `df6076e1ae605f425a380d282c4f25b845a3165f`; integration with current main leaves these documented call paths unchanged. Fixes ENT-3561.
This commit is contained in:
@@ -26,6 +26,26 @@ class BlobReader(io.RawIOBase):
|
||||
|
||||
For local calls, construct a reader from a range-reading callable and the
|
||||
blob's size. The callable accepts an offset and length and returns bytes.
|
||||
This is also the interface for a reader backed by another engine.
|
||||
|
||||
Examples
|
||||
--------
|
||||
```python
|
||||
payload = b"hello world"
|
||||
|
||||
def read_range(offset, length):
|
||||
return payload[offset : offset + length]
|
||||
|
||||
with BlobReader(read_range, len(payload)) as reader:
|
||||
assert reader.read_range(6, 5) == b"world"
|
||||
assert reader.tell() == 0
|
||||
assert reader.read(5) == b"hello"
|
||||
assert reader.tell() == 5
|
||||
```
|
||||
|
||||
``read_range`` does not change the sequential cursor. Use ``size()`` for
|
||||
the declared payload size and close the reader when done; a local reader
|
||||
owns no server state.
|
||||
"""
|
||||
|
||||
def __init__(self, read_range: Callable[[int, int], bytes], size: int):
|
||||
|
||||
@@ -1957,14 +1957,20 @@ def _class_call(cls: type) -> Callable[..., Any]:
|
||||
class UdfDefinition:
|
||||
"""A Python callable prepared for remote Function registration.
|
||||
|
||||
Instances are created with :func:`udf`. Calling an instance executes the
|
||||
original Python function, or constructs the original class, which keeps
|
||||
local unit testing ordinary. Remote execution adapts the callable to the
|
||||
internal Arrow batch ABI described by the registration artifact. By
|
||||
default it invokes the callable once per row. A callable whose parameters
|
||||
are all annotated as ``pyarrow.Array`` receives whole Arrow arrays and
|
||||
returns a ``pyarrow.Array`` instead. This authoring choice is independent
|
||||
of the internal artifact adapter in the registration request.
|
||||
Instances are created with :func:`udf`. Calling an instance is ordinary
|
||||
local Python: a decorated function is called directly, while a decorated
|
||||
class is constructed using its initialization fields and the returned
|
||||
instance is called with each input. Reuse the instance and call its
|
||||
``close`` method when it defines one. This public call does not create a
|
||||
remote application or build the declared remote runtime, and it does not
|
||||
need the private implementation attribute.
|
||||
|
||||
Remote execution adapts the callable to the internal Arrow batch ABI
|
||||
described by the registration artifact. By default it invokes the callable
|
||||
once per row. A callable whose parameters are all annotated as
|
||||
``pyarrow.Array`` receives whole Arrow arrays and returns a
|
||||
``pyarrow.Array`` instead. This authoring choice is independent of the
|
||||
internal artifact adapter in the registration request.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
@@ -2390,8 +2396,14 @@ def udf(
|
||||
... self.factor, self.offset = factor, offset
|
||||
... def __call__(self, value: float) -> float:
|
||||
... return value * self.factor + self.offset
|
||||
>>> scale(factor=2.0)(1.5)
|
||||
... def close(self):
|
||||
... self.factor = 0.0
|
||||
>>> scale_fn = scale(factor=2.0)
|
||||
>>> scale_fn(1.5)
|
||||
3.0
|
||||
>>> scale_fn(2.0)
|
||||
4.0
|
||||
>>> scale_fn.close()
|
||||
>>> [field.name for field in scale.registration_request.signature.initialization]
|
||||
['factor', 'offset']
|
||||
>>> import lancedb
|
||||
@@ -2402,6 +2414,27 @@ def udf(
|
||||
... def copy_blobs(image: pa.Array) -> pa.Array:
|
||||
... return image
|
||||
|
||||
**Local execution.** A function definition is callable with its decorated
|
||||
input signature and runs the original function in the current Python
|
||||
environment. A class definition is called with its initialization
|
||||
arguments and constructs one instance; call that instance for row inputs,
|
||||
reuse it as needed, and call ``close`` if it defines that method. This path
|
||||
does not create a registered
|
||||
[FunctionVersion][lancedb.functions.FunctionVersion], build the declared
|
||||
remote environment, resolve Secrets, or run the remote Arrow adapter or
|
||||
sandbox. A registered ``FunctionVersion`` call is different: it binds
|
||||
``col(...)`` inputs and returns a remote
|
||||
[FunctionApplication][lancedb.functions.FunctionApplication].
|
||||
|
||||
```python
|
||||
scaled = scale(factor=2.0)
|
||||
try:
|
||||
assert scaled(1.5) == 3.0
|
||||
assert scaled(2.0) == 4.0
|
||||
finally:
|
||||
scaled.close()
|
||||
```
|
||||
|
||||
Blob v2 input is exposed as ``pyarrow.LargeBinaryArray`` in vectorized
|
||||
mode (and as ``bytes`` in row mode). Return a ``LargeBinaryArray`` with the
|
||||
declared nullability. Use :func:`lancedb.blob` in the explicit schemas;
|
||||
|
||||
Reference in New Issue
Block a user