docs: explain managed blob ingestion, functions, and cleanup (#4604)

Document explicit Managed Blob v2 opt-in, descriptor queries and payload
fetching, Function inputs and outputs, and whole-object cleanup. Explain
null versus empty values, snapshot ownership, packed-object retention,
and the distinction from the legacy large-binary Blob API.

The local ingestion and Enterprise Function examples have been executed
end to end. The Function lifecycle description depends on
lance-format/lance#10176 and the corresponding Enterprise migration. The
existing Functions page remains outside the published navigation until
launch.

Validation: the 347-file assembled site has no broken links. Mintlify
4.2.888 build validation reports four existing
`RsExampleBody`/`RsExampleTail` import warnings in the reranking and
vector-search pages; the same diagnostics reproduce with the unchanged
base versions of all three pages edited here.
This commit is contained in:
Xuanwo
2026-10-20 22:51:42 +08:00
committed by GitHub
parent 4a03859b2a
commit 9f616e6218
3 changed files with 172 additions and 5 deletions
+25 -4
View File
@@ -65,13 +65,34 @@ files, so a tag on what was the latest version can become an older one by the
time cleanup runs. `optimize()` then raises after compacting, and its index
update does not run.
A deployment's background cleanup follows the same rules. Its own background
jobs, such as compaction and index updates, also commit new versions, so a tag
on what is now the latest version can become an older one without a call from
you. If you rely on reading old versions, set the window to cover the range you
Enterprise background cleanup keeps tagged snapshots while pruning unrelated
expired versions. If you rely on reading untagged old versions, set the window
to cover the range you
need before running this, not after.
</Warning>
## Managed blob objects {#managed-blobs}
[Managed blobs](/management/table#managed-blobs) store out-of-line payloads
under the table's `_blobs/` directory. Cleanup follows the references from
retained snapshots, including protected tags and branches, before selecting
objects for deletion. Replacing a blob or deleting a row makes its old payload
eligible only after no retained snapshot references it.
Cleanup removes whole objects. If a packed object contains several blobs, it
remains while any of them is live. Compaction preserves Managed payload objects
rather than copying their bytes, so compaction alone does not reclaim the
unused portion of a packed object.
Use table maintenance to remove Managed objects; deleting `_blobs/` files
directly can corrupt current or historical snapshots. Upgrade all clients and
maintenance workers to a version that understands Managed blobs before using
them, and reopen handles created before that upgrade.
For local tables, `optimize()` performs compaction and cleanup. Enterprise runs
these as background jobs; the remote Python table does not support `optimize()`.
See [Background jobs](/maintenance/background-jobs) for the Enterprise path.
## Disk usage does not drop immediately {#disk}
Compaction writes new, merged files before the old ones are removed, so disk
+62
View File
@@ -143,6 +143,68 @@ rows added since the last one and recomputes rows whose inputs changed, and
leaves filled rows with unchanged inputs alone. See
[Refresh a column](/feature-engineering/refresh-column).
## Read and produce blobs {#blobs}
A Function can bind a [Blob v2 column](/management/table#managed-blobs) and
return another blob. In row mode, a top-level blob parameter arrives as Python
`bytes`, or `None` for a nullable input. The worker reads the payload before
calling your function. Return bytes and let the destination table store the
Managed payload and descriptor.
Declare blob inputs and outputs explicitly. This example reverses the bytes
to demonstrate the round trip without requiring an image library. Here `db`
is an Enterprise database and `table` contains an `image` Blob v2 column
with table schema metadata `lance.blob.managed=true`, as in the
[table creation example](/management/table#managed-blobs-read-write):
```python
from datetime import timedelta
import lancedb
from lancedb import col
from lancedb.functions import udf
import pyarrow as pa
@udf(
name="reverse_image",
python_version="3.12",
pip=["pyarrow==26.0.0"],
input_schema=pa.schema([lancedb.blob("image")]),
output_schema=lancedb.blob("reversed", nullable=False),
)
def reverse_image(image):
return b"missing" if image is None else image[::-1]
function = db.create_function_async(reverse_image).wait(
timeout=timedelta(minutes=15)
)
table.add_columns({"reversed": function(image=col("image"))})
result = table.refresh_column_async("reversed").wait(
timeout=timedelta(minutes=15)
)
assert result.rows_failed == 0
hits = table.search().select(["id", "reversed"]).with_row_id(True).to_arrow()
values = table.fetch_blobs("reversed", hits).to_pylist()
```
An empty byte string is a filled result. A scalar output cannot be nullable,
because a physical NULL marks an unfilled computed row. To return a nullable
blob as a value, use a named-struct output schema with a nullable blob field;
the struct itself records whether the result has been filled. Read a nested
blob with `table.fetch_blobs("payload.image", hits)`.
On deployments that support computed-column inputs, another Function can bind
to `col("reversed")`. Refresh the producer first, then the consumer. Once
outputs are current, a later refresh fills appended rows. A filtered refresh
preserves payload references for unselected Managed outputs. An unfiltered
refresh can recompute a whole fragment when it cannot establish that the
fragment's outputs are current.
Function output blobs follow the same compaction and cleanup lifecycle as
ingested Managed blobs. Retained versions continue to own their payloads;
cleanup reclaims an object only after its retained references are gone.
## Declare the environment {#environment}
The function runs remotely, so its dependencies are part of its definition:
+85 -1
View File
@@ -427,7 +427,91 @@ Once you have the bytes back from the search result, you can decode them into th
</CodeBlock>
</CodeGroup>
### Large Blobs (Blob API) {#multimodal-large-blobs-blob-api}
### Managed blobs (Blob v2) {#managed-blobs}
Use `lancedb.blob()` for images, audio, videos, and other binary values whose
payloads should load separately from the rest of a row. Write Python `bytes`
normally. Set the table schema metadata `lance.blob.managed` to `true` to
store small values inline and larger values in managed objects under the table's `_blobs/` directory. Medium values can share a packed
object; large values use a dedicated object.
The table owns these objects. You do not upload them separately, keep an object
catalog, or delete their files yourself. A descriptor records where to read the
bytes. Compaction preserves the descriptor and its payload object, so merging
table fragments does not rewrite the blob payloads.
<Warning>
Use an SDK and Enterprise deployment that support Blob v2 Managed storage.
Upgrade every reader, writer, and maintenance worker before activating Managed
blobs on a table, and reopen existing table handles after the upgrade. Older
clients cannot safely maintain a table containing Managed blobs.
</Warning>
#### Write and read blobs {#managed-blobs-read-write}
This example uses a local connection. The same schema, `add`, query, and
`fetch_blobs` calls work with an Enterprise database connection:
```python
import lancedb
import pyarrow as pa
db = lancedb.connect("./media")
table = db.create_table(
"images",
schema=pa.schema([
pa.field("id", pa.int64(), nullable=False),
lancedb.blob("image"),
], metadata={"lance.blob.managed": "true"}),
)
table.add([
{"id": 1, "image": b""},
{"id": 2, "image": bytes(range(256)) * 400},
{"id": 3, "image": None},
])
# Query descriptors first, then fetch only the selected rows' payloads.
hits = (
table.search()
.where("id <= 2")
.select(["id", "image"])
.with_row_id(True)
.to_arrow()
)
images = table.fetch_blobs("image", hits).to_pylist()
by_id = dict(zip(hits["id"].to_pylist(), images, strict=True))
assert by_id[1] == b""
assert len(by_id[2]) == 102400
```
`None` is a null value; `b""` is a present, empty blob. `fetch_blobs` returns
bytes in the order of the supplied row IDs. It materializes the selected
payloads, so fetch bounded batches when blobs are large. Row IDs belong to
their table; fetch them from the same table and snapshot that produced the
query result.
Keep descriptors opaque in application code. Their representation can differ
between inline values, Managed objects, and externally referenced blobs. For
streaming and range reads, see the
[Lance blob guide](https://lance.org/guide/blob/).
#### Derived blobs and maintenance {#managed-blobs-lifecycle}
Enterprise Functions can consume blob columns and
return new blobs. Their output payloads are managed by the destination table.
Local tables can also derive blob columns with
[SQL computed columns](/feature-engineering/add-computed-column).
Deleting a row or replacing its blob does not immediately remove the old
payload: retained table versions may still reference it. [Cleanup](/maintenance/cleanup#managed-blobs)
reclaims objects after those references expire. A packed object remains while
any of its contained blobs is still referenced.
### Legacy large-binary blobs {#multimodal-large-blobs-blob-api}
The schema below describes the earlier large-binary Blob API. It is distinct
from the `lancedb.blob()` schema above; changing the field's metadata alone does
not migrate an existing column to Blob v2.
For larger files like high-resolution images or videos, Lance provides a specialized **Blob API**. By using a large-binary Arrow type (`pa.large_binary()` in Python, `LargeBinary` in TypeScript, and `DataType::LargeBinary` in Rust) and specific metadata, you enable **lazy loading** and optimized encoding. This allows you to work with massive datasets without loading all binary data into memory upfront.