mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-22 03:32:28 +00:00
docs: explain managed blob ingestion, functions, and cleanup (#4604)
Document explicit Managed Blob v2 opt-in, descriptor queries and payload fetching, Function inputs and outputs, and whole-object cleanup. Explain null versus empty values, snapshot ownership, packed-object retention, and the distinction from the legacy large-binary Blob API. The local ingestion and Enterprise Function examples have been executed end to end. The Function lifecycle description depends on lance-format/lance#10176 and the corresponding Enterprise migration. The existing Functions page remains outside the published navigation until launch. Validation: the 347-file assembled site has no broken links. Mintlify 4.2.888 build validation reports four existing `RsExampleBody`/`RsExampleTail` import warnings in the reranking and vector-search pages; the same diagnostics reproduce with the unchanged base versions of all three pages edited here.
This commit is contained in:
@@ -65,13 +65,34 @@ files, so a tag on what was the latest version can become an older one by the
|
||||
time cleanup runs. `optimize()` then raises after compacting, and its index
|
||||
update does not run.
|
||||
|
||||
A deployment's background cleanup follows the same rules. Its own background
|
||||
jobs, such as compaction and index updates, also commit new versions, so a tag
|
||||
on what is now the latest version can become an older one without a call from
|
||||
you. If you rely on reading old versions, set the window to cover the range you
|
||||
Enterprise background cleanup keeps tagged snapshots while pruning unrelated
|
||||
expired versions. If you rely on reading untagged old versions, set the window
|
||||
to cover the range you
|
||||
need before running this, not after.
|
||||
</Warning>
|
||||
|
||||
## Managed blob objects {#managed-blobs}
|
||||
|
||||
[Managed blobs](/management/table#managed-blobs) store out-of-line payloads
|
||||
under the table's `_blobs/` directory. Cleanup follows the references from
|
||||
retained snapshots, including protected tags and branches, before selecting
|
||||
objects for deletion. Replacing a blob or deleting a row makes its old payload
|
||||
eligible only after no retained snapshot references it.
|
||||
|
||||
Cleanup removes whole objects. If a packed object contains several blobs, it
|
||||
remains while any of them is live. Compaction preserves Managed payload objects
|
||||
rather than copying their bytes, so compaction alone does not reclaim the
|
||||
unused portion of a packed object.
|
||||
|
||||
Use table maintenance to remove Managed objects; deleting `_blobs/` files
|
||||
directly can corrupt current or historical snapshots. Upgrade all clients and
|
||||
maintenance workers to a version that understands Managed blobs before using
|
||||
them, and reopen handles created before that upgrade.
|
||||
|
||||
For local tables, `optimize()` performs compaction and cleanup. Enterprise runs
|
||||
these as background jobs; the remote Python table does not support `optimize()`.
|
||||
See [Background jobs](/maintenance/background-jobs) for the Enterprise path.
|
||||
|
||||
## Disk usage does not drop immediately {#disk}
|
||||
|
||||
Compaction writes new, merged files before the old ones are removed, so disk
|
||||
|
||||
@@ -143,6 +143,68 @@ rows added since the last one and recomputes rows whose inputs changed, and
|
||||
leaves filled rows with unchanged inputs alone. See
|
||||
[Refresh a column](/feature-engineering/refresh-column).
|
||||
|
||||
## Read and produce blobs {#blobs}
|
||||
|
||||
A Function can bind a [Blob v2 column](/management/table#managed-blobs) and
|
||||
return another blob. In row mode, a top-level blob parameter arrives as Python
|
||||
`bytes`, or `None` for a nullable input. The worker reads the payload before
|
||||
calling your function. Return bytes and let the destination table store the
|
||||
Managed payload and descriptor.
|
||||
|
||||
Declare blob inputs and outputs explicitly. This example reverses the bytes
|
||||
to demonstrate the round trip without requiring an image library. Here `db`
|
||||
is an Enterprise database and `table` contains an `image` Blob v2 column
|
||||
with table schema metadata `lance.blob.managed=true`, as in the
|
||||
[table creation example](/management/table#managed-blobs-read-write):
|
||||
|
||||
```python
|
||||
from datetime import timedelta
|
||||
|
||||
import lancedb
|
||||
from lancedb import col
|
||||
from lancedb.functions import udf
|
||||
import pyarrow as pa
|
||||
|
||||
@udf(
|
||||
name="reverse_image",
|
||||
python_version="3.12",
|
||||
pip=["pyarrow==26.0.0"],
|
||||
input_schema=pa.schema([lancedb.blob("image")]),
|
||||
output_schema=lancedb.blob("reversed", nullable=False),
|
||||
)
|
||||
def reverse_image(image):
|
||||
return b"missing" if image is None else image[::-1]
|
||||
|
||||
function = db.create_function_async(reverse_image).wait(
|
||||
timeout=timedelta(minutes=15)
|
||||
)
|
||||
table.add_columns({"reversed": function(image=col("image"))})
|
||||
result = table.refresh_column_async("reversed").wait(
|
||||
timeout=timedelta(minutes=15)
|
||||
)
|
||||
assert result.rows_failed == 0
|
||||
|
||||
hits = table.search().select(["id", "reversed"]).with_row_id(True).to_arrow()
|
||||
values = table.fetch_blobs("reversed", hits).to_pylist()
|
||||
```
|
||||
|
||||
An empty byte string is a filled result. A scalar output cannot be nullable,
|
||||
because a physical NULL marks an unfilled computed row. To return a nullable
|
||||
blob as a value, use a named-struct output schema with a nullable blob field;
|
||||
the struct itself records whether the result has been filled. Read a nested
|
||||
blob with `table.fetch_blobs("payload.image", hits)`.
|
||||
|
||||
On deployments that support computed-column inputs, another Function can bind
|
||||
to `col("reversed")`. Refresh the producer first, then the consumer. Once
|
||||
outputs are current, a later refresh fills appended rows. A filtered refresh
|
||||
preserves payload references for unselected Managed outputs. An unfiltered
|
||||
refresh can recompute a whole fragment when it cannot establish that the
|
||||
fragment's outputs are current.
|
||||
|
||||
Function output blobs follow the same compaction and cleanup lifecycle as
|
||||
ingested Managed blobs. Retained versions continue to own their payloads;
|
||||
cleanup reclaims an object only after its retained references are gone.
|
||||
|
||||
## Declare the environment {#environment}
|
||||
|
||||
The function runs remotely, so its dependencies are part of its definition:
|
||||
|
||||
@@ -427,7 +427,91 @@ Once you have the bytes back from the search result, you can decode them into th
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
### Large Blobs (Blob API) {#multimodal-large-blobs-blob-api}
|
||||
### Managed blobs (Blob v2) {#managed-blobs}
|
||||
|
||||
Use `lancedb.blob()` for images, audio, videos, and other binary values whose
|
||||
payloads should load separately from the rest of a row. Write Python `bytes`
|
||||
normally. Set the table schema metadata `lance.blob.managed` to `true` to
|
||||
store small values inline and larger values in managed objects under the table's `_blobs/` directory. Medium values can share a packed
|
||||
object; large values use a dedicated object.
|
||||
|
||||
The table owns these objects. You do not upload them separately, keep an object
|
||||
catalog, or delete their files yourself. A descriptor records where to read the
|
||||
bytes. Compaction preserves the descriptor and its payload object, so merging
|
||||
table fragments does not rewrite the blob payloads.
|
||||
|
||||
<Warning>
|
||||
Use an SDK and Enterprise deployment that support Blob v2 Managed storage.
|
||||
Upgrade every reader, writer, and maintenance worker before activating Managed
|
||||
blobs on a table, and reopen existing table handles after the upgrade. Older
|
||||
clients cannot safely maintain a table containing Managed blobs.
|
||||
</Warning>
|
||||
|
||||
#### Write and read blobs {#managed-blobs-read-write}
|
||||
|
||||
This example uses a local connection. The same schema, `add`, query, and
|
||||
`fetch_blobs` calls work with an Enterprise database connection:
|
||||
|
||||
```python
|
||||
import lancedb
|
||||
import pyarrow as pa
|
||||
|
||||
db = lancedb.connect("./media")
|
||||
table = db.create_table(
|
||||
"images",
|
||||
schema=pa.schema([
|
||||
pa.field("id", pa.int64(), nullable=False),
|
||||
lancedb.blob("image"),
|
||||
], metadata={"lance.blob.managed": "true"}),
|
||||
)
|
||||
table.add([
|
||||
{"id": 1, "image": b""},
|
||||
{"id": 2, "image": bytes(range(256)) * 400},
|
||||
{"id": 3, "image": None},
|
||||
])
|
||||
|
||||
# Query descriptors first, then fetch only the selected rows' payloads.
|
||||
hits = (
|
||||
table.search()
|
||||
.where("id <= 2")
|
||||
.select(["id", "image"])
|
||||
.with_row_id(True)
|
||||
.to_arrow()
|
||||
)
|
||||
images = table.fetch_blobs("image", hits).to_pylist()
|
||||
by_id = dict(zip(hits["id"].to_pylist(), images, strict=True))
|
||||
assert by_id[1] == b""
|
||||
assert len(by_id[2]) == 102400
|
||||
```
|
||||
|
||||
`None` is a null value; `b""` is a present, empty blob. `fetch_blobs` returns
|
||||
bytes in the order of the supplied row IDs. It materializes the selected
|
||||
payloads, so fetch bounded batches when blobs are large. Row IDs belong to
|
||||
their table; fetch them from the same table and snapshot that produced the
|
||||
query result.
|
||||
|
||||
Keep descriptors opaque in application code. Their representation can differ
|
||||
between inline values, Managed objects, and externally referenced blobs. For
|
||||
streaming and range reads, see the
|
||||
[Lance blob guide](https://lance.org/guide/blob/).
|
||||
|
||||
#### Derived blobs and maintenance {#managed-blobs-lifecycle}
|
||||
|
||||
Enterprise Functions can consume blob columns and
|
||||
return new blobs. Their output payloads are managed by the destination table.
|
||||
Local tables can also derive blob columns with
|
||||
[SQL computed columns](/feature-engineering/add-computed-column).
|
||||
|
||||
Deleting a row or replacing its blob does not immediately remove the old
|
||||
payload: retained table versions may still reference it. [Cleanup](/maintenance/cleanup#managed-blobs)
|
||||
reclaims objects after those references expire. A packed object remains while
|
||||
any of its contained blobs is still referenced.
|
||||
|
||||
### Legacy large-binary blobs {#multimodal-large-blobs-blob-api}
|
||||
|
||||
The schema below describes the earlier large-binary Blob API. It is distinct
|
||||
from the `lancedb.blob()` schema above; changing the field's metadata alone does
|
||||
not migrate an existing column to Blob v2.
|
||||
|
||||
For larger files like high-resolution images or videos, Lance provides a specialized **Blob API**. By using a large-binary Arrow type (`pa.large_binary()` in Python, `LargeBinary` in TypeScript, and `DataType::LargeBinary` in Rust) and specific metadata, you enable **lazy loading** and optimized encoding. This allows you to work with massive datasets without loading all binary data into memory upfront.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user