From 9f616e6218873333a95208950350cca75a526aa8 Mon Sep 17 00:00:00 2001 From: Xuanwo Date: Tue, 20 Oct 2026 22:51:42 +0800 Subject: [PATCH] docs: explain managed blob ingestion, functions, and cleanup (#4604) Document explicit Managed Blob v2 opt-in, descriptor queries and payload fetching, Function inputs and outputs, and whole-object cleanup. Explain null versus empty values, snapshot ownership, packed-object retention, and the distinction from the legacy large-binary Blob API. The local ingestion and Enterprise Function examples have been executed end to end. The Function lifecycle description depends on lance-format/lance#10176 and the corresponding Enterprise migration. The existing Functions page remains outside the published navigation until launch. Validation: the 347-file assembled site has no broken links. Mintlify 4.2.888 build validation reports four existing `RsExampleBody`/`RsExampleTail` import warnings in the reranking and vector-search pages; the same diagnostics reproduce with the unchanged base versions of all three pages edited here. --- docs/web/maintenance/cleanup.mdx | 29 +++++++++-- docs/web/management/function.mdx | 62 +++++++++++++++++++++++ docs/web/management/table.mdx | 86 +++++++++++++++++++++++++++++++- 3 files changed, 172 insertions(+), 5 deletions(-) diff --git a/docs/web/maintenance/cleanup.mdx b/docs/web/maintenance/cleanup.mdx index 397def690..7833fa8a0 100644 --- a/docs/web/maintenance/cleanup.mdx +++ b/docs/web/maintenance/cleanup.mdx @@ -65,13 +65,34 @@ files, so a tag on what was the latest version can become an older one by the time cleanup runs. `optimize()` then raises after compacting, and its index update does not run. -A deployment's background cleanup follows the same rules. Its own background -jobs, such as compaction and index updates, also commit new versions, so a tag -on what is now the latest version can become an older one without a call from -you. If you rely on reading old versions, set the window to cover the range you +Enterprise background cleanup keeps tagged snapshots while pruning unrelated +expired versions. If you rely on reading untagged old versions, set the window +to cover the range you need before running this, not after. +## Managed blob objects {#managed-blobs} + +[Managed blobs](/management/table#managed-blobs) store out-of-line payloads +under the table's `_blobs/` directory. Cleanup follows the references from +retained snapshots, including protected tags and branches, before selecting +objects for deletion. Replacing a blob or deleting a row makes its old payload +eligible only after no retained snapshot references it. + +Cleanup removes whole objects. If a packed object contains several blobs, it +remains while any of them is live. Compaction preserves Managed payload objects +rather than copying their bytes, so compaction alone does not reclaim the +unused portion of a packed object. + +Use table maintenance to remove Managed objects; deleting `_blobs/` files +directly can corrupt current or historical snapshots. Upgrade all clients and +maintenance workers to a version that understands Managed blobs before using +them, and reopen handles created before that upgrade. + +For local tables, `optimize()` performs compaction and cleanup. Enterprise runs +these as background jobs; the remote Python table does not support `optimize()`. +See [Background jobs](/maintenance/background-jobs) for the Enterprise path. + ## Disk usage does not drop immediately {#disk} Compaction writes new, merged files before the old ones are removed, so disk diff --git a/docs/web/management/function.mdx b/docs/web/management/function.mdx index 4467c6abb..8e5b53e4a 100644 --- a/docs/web/management/function.mdx +++ b/docs/web/management/function.mdx @@ -143,6 +143,68 @@ rows added since the last one and recomputes rows whose inputs changed, and leaves filled rows with unchanged inputs alone. See [Refresh a column](/feature-engineering/refresh-column). +## Read and produce blobs {#blobs} + +A Function can bind a [Blob v2 column](/management/table#managed-blobs) and +return another blob. In row mode, a top-level blob parameter arrives as Python +`bytes`, or `None` for a nullable input. The worker reads the payload before +calling your function. Return bytes and let the destination table store the +Managed payload and descriptor. + +Declare blob inputs and outputs explicitly. This example reverses the bytes +to demonstrate the round trip without requiring an image library. Here `db` +is an Enterprise database and `table` contains an `image` Blob v2 column +with table schema metadata `lance.blob.managed=true`, as in the +[table creation example](/management/table#managed-blobs-read-write): + +```python +from datetime import timedelta + +import lancedb +from lancedb import col +from lancedb.functions import udf +import pyarrow as pa + +@udf( + name="reverse_image", + python_version="3.12", + pip=["pyarrow==26.0.0"], + input_schema=pa.schema([lancedb.blob("image")]), + output_schema=lancedb.blob("reversed", nullable=False), +) +def reverse_image(image): + return b"missing" if image is None else image[::-1] + +function = db.create_function_async(reverse_image).wait( + timeout=timedelta(minutes=15) +) +table.add_columns({"reversed": function(image=col("image"))}) +result = table.refresh_column_async("reversed").wait( + timeout=timedelta(minutes=15) +) +assert result.rows_failed == 0 + +hits = table.search().select(["id", "reversed"]).with_row_id(True).to_arrow() +values = table.fetch_blobs("reversed", hits).to_pylist() +``` + +An empty byte string is a filled result. A scalar output cannot be nullable, +because a physical NULL marks an unfilled computed row. To return a nullable +blob as a value, use a named-struct output schema with a nullable blob field; +the struct itself records whether the result has been filled. Read a nested +blob with `table.fetch_blobs("payload.image", hits)`. + +On deployments that support computed-column inputs, another Function can bind +to `col("reversed")`. Refresh the producer first, then the consumer. Once +outputs are current, a later refresh fills appended rows. A filtered refresh +preserves payload references for unselected Managed outputs. An unfiltered +refresh can recompute a whole fragment when it cannot establish that the +fragment's outputs are current. + +Function output blobs follow the same compaction and cleanup lifecycle as +ingested Managed blobs. Retained versions continue to own their payloads; +cleanup reclaims an object only after its retained references are gone. + ## Declare the environment {#environment} The function runs remotely, so its dependencies are part of its definition: diff --git a/docs/web/management/table.mdx b/docs/web/management/table.mdx index df44a8fd3..ed9883277 100644 --- a/docs/web/management/table.mdx +++ b/docs/web/management/table.mdx @@ -427,7 +427,91 @@ Once you have the bytes back from the search result, you can decode them into th -### Large Blobs (Blob API) {#multimodal-large-blobs-blob-api} +### Managed blobs (Blob v2) {#managed-blobs} + +Use `lancedb.blob()` for images, audio, videos, and other binary values whose +payloads should load separately from the rest of a row. Write Python `bytes` +normally. Set the table schema metadata `lance.blob.managed` to `true` to +store small values inline and larger values in managed objects under the table's `_blobs/` directory. Medium values can share a packed +object; large values use a dedicated object. + +The table owns these objects. You do not upload them separately, keep an object +catalog, or delete their files yourself. A descriptor records where to read the +bytes. Compaction preserves the descriptor and its payload object, so merging +table fragments does not rewrite the blob payloads. + + +Use an SDK and Enterprise deployment that support Blob v2 Managed storage. +Upgrade every reader, writer, and maintenance worker before activating Managed +blobs on a table, and reopen existing table handles after the upgrade. Older +clients cannot safely maintain a table containing Managed blobs. + + +#### Write and read blobs {#managed-blobs-read-write} + +This example uses a local connection. The same schema, `add`, query, and +`fetch_blobs` calls work with an Enterprise database connection: + +```python +import lancedb +import pyarrow as pa + +db = lancedb.connect("./media") +table = db.create_table( + "images", + schema=pa.schema([ + pa.field("id", pa.int64(), nullable=False), + lancedb.blob("image"), + ], metadata={"lance.blob.managed": "true"}), +) +table.add([ + {"id": 1, "image": b""}, + {"id": 2, "image": bytes(range(256)) * 400}, + {"id": 3, "image": None}, +]) + +# Query descriptors first, then fetch only the selected rows' payloads. +hits = ( + table.search() + .where("id <= 2") + .select(["id", "image"]) + .with_row_id(True) + .to_arrow() +) +images = table.fetch_blobs("image", hits).to_pylist() +by_id = dict(zip(hits["id"].to_pylist(), images, strict=True)) +assert by_id[1] == b"" +assert len(by_id[2]) == 102400 +``` + +`None` is a null value; `b""` is a present, empty blob. `fetch_blobs` returns +bytes in the order of the supplied row IDs. It materializes the selected +payloads, so fetch bounded batches when blobs are large. Row IDs belong to +their table; fetch them from the same table and snapshot that produced the +query result. + +Keep descriptors opaque in application code. Their representation can differ +between inline values, Managed objects, and externally referenced blobs. For +streaming and range reads, see the +[Lance blob guide](https://lance.org/guide/blob/). + +#### Derived blobs and maintenance {#managed-blobs-lifecycle} + +Enterprise Functions can consume blob columns and +return new blobs. Their output payloads are managed by the destination table. +Local tables can also derive blob columns with +[SQL computed columns](/feature-engineering/add-computed-column). + +Deleting a row or replacing its blob does not immediately remove the old +payload: retained table versions may still reference it. [Cleanup](/maintenance/cleanup#managed-blobs) +reclaims objects after those references expire. A packed object remains while +any of its contained blobs is still referenced. + +### Legacy large-binary blobs {#multimodal-large-blobs-blob-api} + +The schema below describes the earlier large-binary Blob API. It is distinct +from the `lancedb.blob()` schema above; changing the field's metadata alone does +not migrate an existing column to Blob v2. For larger files like high-resolution images or videos, Lance provides a specialized **Blob API**. By using a large-binary Arrow type (`pa.large_binary()` in Python, `LargeBinary` in TypeScript, and `DataType::LargeBinary` in Rust) and specific metadata, you enable **lazy loading** and optimized encoding. This allows you to work with massive datasets without loading all binary data into memory upfront.