mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-27 14:06:14 +00:00
docs: document Blob V2 in the multimodal guide (#4623)
This commit is contained in:
+370
-111
@@ -110,15 +110,40 @@ import {
|
||||
PyProcessResults as ProcessResults,
|
||||
TsProcessResults as TsProcessResults,
|
||||
RsProcessResults as RsProcessResults,
|
||||
PyBlobApiSetup as BlobApiSetup,
|
||||
TsBlobApiSetup as TsBlobApiSetup,
|
||||
RsBlobApiImports as RsBlobApiImports,
|
||||
RsBlobApiSetup as RsBlobApiSetup,
|
||||
PyBlobApiSchema as BlobApiSchema,
|
||||
TsBlobApiSchema as TsBlobApiSchema,
|
||||
RsBlobApiSchema as RsBlobApiSchema,
|
||||
PyBlobApiIngest as BlobApiIngest,
|
||||
PyBlobApiBytes as BlobApiBytes,
|
||||
TsBlobApiIngest as TsBlobApiIngest,
|
||||
RsBlobApiIngest as RsBlobApiIngest,
|
||||
PyBlobApiFiles as BlobApiFiles,
|
||||
TsBlobApiFiles as TsBlobApiFiles,
|
||||
RsBlobApiFiles as RsBlobApiFiles,
|
||||
PyBlobApiFetch as BlobApiFetch,
|
||||
TsBlobApiFetch as TsBlobApiFetch,
|
||||
RsBlobApiFetch as RsBlobApiFetch,
|
||||
PyBlobApiRanges as BlobApiRanges,
|
||||
TsBlobApiRanges as TsBlobApiRanges,
|
||||
RsBlobApiRanges as RsBlobApiRanges,
|
||||
TsBlobApiBytesMode as TsBlobApiBytesMode,
|
||||
PyLegacyBlobApiSchema as LegacyBlobApiSchema,
|
||||
TsLegacyBlobApiSchema as TsLegacyBlobApiSchema,
|
||||
RsLegacyBlobApiSchema as RsLegacyBlobApiSchema,
|
||||
PyLegacyBlobApiIngest as LegacyBlobApiIngest,
|
||||
TsLegacyBlobApiIngest as TsLegacyBlobApiIngest,
|
||||
RsLegacyBlobApiIngest as RsLegacyBlobApiIngest,
|
||||
PyLegacyBlobApiToPandas as LegacyBlobApiToPandas,
|
||||
PyLegacyQueryToPandasKwargs as LegacyQueryToPandasKwargs,
|
||||
} from '/snippets/multimodal.mdx';
|
||||
import {
|
||||
PyBlobApiToPandas as BlobApiToPandas,
|
||||
PyQueryToPandasKwargs as QueryToPandasKwargs,
|
||||
} from '/snippets/multimodal.mdx';
|
||||
} from '/snippets/blob_pylance.mdx';
|
||||
|
||||
A table stores data in rows and columns and keeps a history of committed
|
||||
versions. Queries read table data, writes update it, and indexes accelerate
|
||||
@@ -307,138 +332,280 @@ extension types and pseudo-columns.
|
||||
|
||||
## Multimodal data {#multimodal}
|
||||
|
||||
Binary data (images, audio, video, model artifacts) lives in the table
|
||||
beside the structured columns and embeddings that describe it, and is governed
|
||||
by the same versioning and search.
|
||||
Store images, audio, video, and documents alongside structured columns and
|
||||
embeddings in the same versioned table.
|
||||
|
||||
Small values go in an ordinary `binary` column. Large ones go in a **blob**
|
||||
column, stored out of line so a scan over the other columns stays fast. See
|
||||
[Data types](/sql/data-types#blobs) for how the two differ to a reader.
|
||||
Use `blob` columns for these payloads. Small values stay inline in the data
|
||||
file, up to 64 KiB by default, and larger values are stored outside it.
|
||||
|
||||
### Store binary data {#multimodal-store-binary-data}
|
||||
### Blob columns {#multimodal-blob-v2}
|
||||
|
||||
To store binary data, define a binary Arrow field in your schema (`pa.binary()` in Python, `Binary` in TypeScript, and `DataType::Binary` in Rust).
|
||||
<Warning>
|
||||
Blob V2 is in early access.
|
||||
</Warning>
|
||||
|
||||
#### 1\. Setup and imports {#multimodal-1-setup-and-imports}
|
||||
A `blob` column keeps metadata separate from the payload bytes, so queries can
|
||||
filter rows without loading the blobs.
|
||||
|
||||
First, import the necessary libraries for LanceDB and Arrow in your SDK.
|
||||
Most queries return descriptors, which contain metadata about the blob instead
|
||||
of its bytes. You can
|
||||
[read the bytes in the query results](#multimodal-blob-query-results) or
|
||||
[fetch them by row ID](#multimodal-read-blobs).
|
||||
|
||||
In Python, plain `to_pandas()` scans can also return `BlobFile` handles when
|
||||
`pylance` is installed.
|
||||
|
||||
When you write raw bytes, Lance chooses a storage layout based on the payload
|
||||
size. A descriptor's `kind` tells you which one was used:
|
||||
|
||||
| Storage mode | `kind` | Default placement | Storage behavior |
|
||||
| --- | --- | --- | --- |
|
||||
| `Inline` | `0` | Up to 64 KiB | Stored in the main data file. |
|
||||
| `Packed` | `1` | Above 64 KiB, up to 4 MiB | Packed with other blobs into a shared sidecar file, up to 1 GiB per file by default. |
|
||||
| `Dedicated` | `2` | Above 4 MiB | Stored in its own raw sidecar file. |
|
||||
| `Managed` | `4` or `5` | Out-of-line values when managed storage is enabled | Stored in a Lance-owned object independently of the data file. |
|
||||
|
||||
Managed storage must be enabled separately. See
|
||||
[Managed blobs](#managed-blobs) for details. `kind=5` identifies a Managed
|
||||
object using an explicitly registered base.
|
||||
|
||||
Lance uses the default thresholds unless you customize them. Explicitly
|
||||
configured thresholds are stored in the table schema and preserved on append.
|
||||
|
||||
#### Setup {#multimodal-blob-v2-setup}
|
||||
|
||||
Import the SDK and open a local database. The examples in this section use `db`.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{MultimodalImports}
|
||||
{BlobApiSetup}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsMultimodalImports}
|
||||
{TsMultimodalImports}{"\n"}{TsBlobApiSetup}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsMultimodalImports}
|
||||
{RsBlobApiImports}{"\n"}{RsBlobApiSetup}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
#### 2\. Prepare data {#multimodal-2-prepare-data}
|
||||
#### Define a blob schema {#multimodal-blob-v2-schema}
|
||||
|
||||
For this example, we'll create some dummy in-memory images. In a real application, you would read these from files or an API. The key is to convert your data (image, audio, etc.) into a raw `bytes` object.
|
||||
Declare a blob column with `lancedb.blob()` in Python and TypeScript, or
|
||||
`blob()` in Rust:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{CreateDummyData}
|
||||
{BlobApiSchema}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsCreateDummyData}
|
||||
{TsBlobApiSchema}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsCreateDummyData}
|
||||
{RsBlobApiSchema}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
#### 3\. Define the schema {#multimodal-3-define-the-schema}
|
||||
#### Ingest blobs {#multimodal-blob-v2-ingest}
|
||||
|
||||
When creating the table, it is **highly recommended** to define the schema explicitly. This ensures that your binary data is correctly interpreted as a `binary` type by Arrow/LanceDB and not as a generic string or list.
|
||||
Pass raw bytes for each blob value. In Python, a blob column accepts `bytes`.
|
||||
In TypeScript, it accepts a `Buffer`. The Python example creates two JPEGs
|
||||
with EXIF metadata, which we'll read using a blob handle below.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{DefineSchema}
|
||||
{BlobApiIngest}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsDefineSchema}
|
||||
{TsBlobApiIngest}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsDefineSchema}
|
||||
{RsBlobApiIngest}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
#### 4\. Ingest data {#multimodal-4-ingest-data}
|
||||
#### Read bytes in query results {#multimodal-blob-query-results}
|
||||
|
||||
Now, create the table using the data and the defined schema.
|
||||
Python and TypeScript can return blob bytes in query results without
|
||||
collecting `_rowid`. Rust uses [`fetch_blobs`](#multimodal-read-blobs) with
|
||||
row IDs instead.
|
||||
|
||||
##### Python {#multimodal-blob-python}
|
||||
|
||||
`to_list(blob_mode="bytes")` returns the `image` bytes from the `photos` table
|
||||
above:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{IngestData}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsIngestData}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsIngestData}
|
||||
{BlobApiBytes}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
### Retrieve and use blobs {#multimodal-retrieve-and-use-blobs}
|
||||
In the Python SDK, `Table.to_pandas()`, `Table.to_arrow()`, and the query builder
|
||||
methods `to_pandas()`, `to_arrow()`, and `to_list()` accept a `blob_mode`
|
||||
argument:
|
||||
|
||||
When you search your LanceDB table, you can retrieve the binary column just like any other metadata.
|
||||
| `blob_mode` | `to_pandas()` | `to_arrow()` and `to_list()` |
|
||||
| --- | --- | --- |
|
||||
| `"lazy"` | Default. Plain scans return seekable `BlobFile` handles. Vector, full-text, hybrid, and ordered queries return descriptor dicts. | Default. Returns descriptors carrying `_lance_row_id` so the result table can be passed to `fetch_blobs` or `fetch_blob_files`. |
|
||||
| `"descriptions"` | Returns descriptor dicts with `kind`, `position`, `size`, `blob_id`, and `blob_uri`. | Returns the 5-field descriptor without `_lance_row_id`. |
|
||||
| `"bytes"` | Materializes each blob as `bytes` in the DataFrame. | Materializes each blob as `large_binary` in Arrow or `bytes` in Python dicts. |
|
||||
|
||||
`to_pandas(blob_mode="lazy")` and `to_pandas(blob_mode="bytes")` require
|
||||
pylance. Without it they raise `ImportError`. `to_arrow()`,
|
||||
`to_list(blob_mode="bytes")`, `to_pandas(blob_mode="descriptions")`, and the
|
||||
fetch methods do not.
|
||||
|
||||
`blob_mode="bytes"` requires a local table snapshot, so remote and in-memory
|
||||
tables raise `NotImplementedError`. On a remote table, `Table.to_pandas()` and
|
||||
`Table.to_arrow()` aren't supported, and a query's `to_pandas()` also raises in
|
||||
the default `"lazy"` mode because it would open `BlobFile` handles. Query with
|
||||
`blob_mode="descriptions"`, then read bytes with `fetch_blobs` or
|
||||
`fetch_blob_files`.
|
||||
|
||||
`blob_mode="bytes"` materializes top-level and struct-nested blob columns. It
|
||||
raises `ValueError` when the selected columns include blobs
|
||||
inside a list. Select those columns with `blob_mode="descriptions"`, or leave
|
||||
them out of the projection. Extra keyword arguments on
|
||||
`to_pandas()`, such as `split_blocks` and `self_destruct`, are forwarded to
|
||||
PyArrow.
|
||||
|
||||
###### Optional pandas examples {#multimodal-blob-pandas}
|
||||
|
||||
These examples require pylance. The query example expects `tbl_async`,
|
||||
`query_vector`, and a `video` column, which the `photos` table does not define.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{SearchData}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsSearchData}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsSearchData}
|
||||
{BlobApiToPandas}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
#### Convert bytes back to objects {#multimodal-convert-bytes-back-to-objects}
|
||||
|
||||
Once you have the bytes back from the search result, you can decode them into the original format (for example, an image object or audio buffer).
|
||||
Sync and async query builders accept the same `blob_mode` values:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{ProcessResults}
|
||||
{QueryToPandasKwargs}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
##### TypeScript {#multimodal-blob-typescript}
|
||||
|
||||
In TypeScript, `toArray()` and `toArrow()` accept a `blobMode` option:
|
||||
|
||||
| `blobMode` | Result |
|
||||
| --- | --- |
|
||||
| `"descriptions"` | Default. Returns descriptors. |
|
||||
| `"bytes"` | Returns each top-level blob as bytes, or `null` for a null blob. Blobs nested in a struct or list keep their descriptors. |
|
||||
|
||||
With `"bytes"`, the rows and the bytes come from the same table version. If the
|
||||
table changes while the query runs, the query runs again, up to three times:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsBlobApiBytesMode}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
#### Read blobs on demand {#multimodal-read-blobs}
|
||||
|
||||
Query the rows you want and pass their `_rowid` values to a fetch method.
|
||||
`fetch_blob_files` opens a seekable handle and loads bytes only when you read.
|
||||
The Python example opens the JPEG from the ingest step and reads its EXIF
|
||||
Make tag with Pillow. TypeScript and Rust read the first bytes from the handle.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{BlobApiFiles}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsProcessResults}
|
||||
{TsBlobApiFiles}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsProcessResults}
|
||||
{RsBlobApiFiles}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
The other fetch methods load bytes directly:
|
||||
|
||||
| Need | Python | TypeScript | Rust |
|
||||
| --- | --- | --- | --- |
|
||||
| Whole payloads for a set of rows | `fetch_blobs` | `fetchBlobs` | `fetch_blobs` |
|
||||
| Byte slices across many rows in one call | `fetch_blob_ranges` | Not available. Use `fetchBlobFiles` and `readRanges`. | `fetch_blob_ranges` |
|
||||
|
||||
- `fetch_blobs` returns the full bytes for each row ID in order. Null blobs
|
||||
produce null entries, and empty blobs produce empty byte strings.
|
||||
- `fetch_blob_files` loads bytes when you call `read`, `read_range`, or
|
||||
`read_ranges`. Range reads don't move the file cursor. Null blobs produce
|
||||
null handles.
|
||||
- `fetch_blob_ranges` takes `(row_id, offset, length)` requests and returns one
|
||||
slice per request, in order. A request list can repeat a row to read more
|
||||
than one slice of that blob.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{BlobApiFetch}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsBlobApiFetch}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsBlobApiFetch}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
Python and Rust read slices across rows with `fetch_blob_ranges`. On a handle,
|
||||
`read_ranges` reads more than one slice of one blob. Python takes
|
||||
`(offset, length)` pairs, TypeScript takes `{ start, end }` ranges with an
|
||||
exclusive `end`, and Rust takes `start..end` ranges:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{BlobApiRanges}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsBlobApiRanges}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsBlobApiRanges}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
#### Notes and limits {#multimodal-blob-notes}
|
||||
|
||||
Blob fetches fail if any requested row ID is missing or deleted. With stable
|
||||
row IDs enabled, IDs remain valid across compaction.
|
||||
|
||||
A blob field can sit at the top level of the schema or inside a struct or list.
|
||||
Struct-nested blob columns use dotted paths such as `clip.video` in the fetch
|
||||
APIs. Blobs inside lists can be written and queried as descriptors. A table
|
||||
row ID does not address one list element, so `blob_columns()`, `fetch_blobs`,
|
||||
`fetch_blob_files`, and `blob_mode="bytes"` can't return those bytes.
|
||||
|
||||
See the Lance [blob guide](https://lance.org/guide/blob/) for the storage format.
|
||||
|
||||
### Managed blobs (Blob v2) {#managed-blobs}
|
||||
|
||||
Use `lancedb.blob()` for images, audio, videos, and other binary values whose
|
||||
payloads should load separately from the rest of a row. Write Python `bytes`
|
||||
normally. Set the table schema metadata `lance.blob.managed` to `true` to
|
||||
store small values inline and larger values in managed objects under the table's `_blobs/` directory. Medium values can share a packed
|
||||
object; large values use a dedicated object.
|
||||
Set the table schema metadata `lance.blob.managed` to `true` on a table with
|
||||
[`lancedb.blob()`](#multimodal-blob-v2) columns to store small values inline and
|
||||
larger values in managed objects under the table's `_blobs/` directory. Medium
|
||||
values can share a packed object, and large values use a dedicated object.
|
||||
|
||||
The table owns these objects. You do not upload them separately, keep an object
|
||||
catalog, or delete their files yourself. A descriptor records where to read the
|
||||
bytes. Compaction preserves the descriptor and its payload object, so merging
|
||||
table fragments does not rewrite the blob payloads.
|
||||
catalog, or delete their files yourself. Compaction preserves each descriptor
|
||||
and its payload object, so merging table fragments does not rewrite the blob
|
||||
payloads.
|
||||
|
||||
<Warning>
|
||||
Use an SDK and Enterprise deployment that support Blob v2 Managed storage.
|
||||
@@ -484,16 +651,10 @@ assert by_id[1] == b""
|
||||
assert len(by_id[2]) == 102400
|
||||
```
|
||||
|
||||
`None` is a null value; `b""` is a present, empty blob. `fetch_blobs` returns
|
||||
bytes in the order of the supplied row IDs. It materializes the selected
|
||||
payloads, so fetch bounded batches when blobs are large. Row IDs belong to
|
||||
their table; fetch them from the same table and snapshot that produced the
|
||||
query result.
|
||||
|
||||
Keep descriptors opaque in application code. Their representation can differ
|
||||
between inline values, Managed objects, and externally referenced blobs. For
|
||||
streaming and range reads, see the
|
||||
[Lance blob guide](https://lance.org/guide/blob/).
|
||||
`fetch_blobs` materializes the selected payloads in memory, so fetch bounded
|
||||
batches when blobs are large. Row IDs belong to their table and snapshot. Keep
|
||||
descriptors opaque in application code, because their representation can differ
|
||||
between inline values and Managed objects.
|
||||
|
||||
#### Derived blobs and maintenance {#managed-blobs-lifecycle}
|
||||
|
||||
@@ -509,92 +670,190 @@ any of its contained blobs is still referenced.
|
||||
|
||||
### Legacy large-binary blobs {#multimodal-large-blobs-blob-api}
|
||||
|
||||
The schema below describes the earlier large-binary Blob API. It is distinct
|
||||
from the `lancedb.blob()` schema above; changing the field's metadata alone does
|
||||
not migrate an existing column to Blob v2.
|
||||
|
||||
For larger files like high-resolution images or videos, Lance provides a specialized **Blob API**. By using a large-binary Arrow type (`pa.large_binary()` in Python, `LargeBinary` in TypeScript, and `DataType::LargeBinary` in Rust) and specific metadata, you enable **lazy loading** and optimized encoding. This allows you to work with massive datasets without loading all binary data into memory upfront.
|
||||
Use [`lancedb.blob()`](#multimodal-blob-v2) for new tables. The sections below
|
||||
cover existing tables that use the V1 large-binary Blob API
|
||||
(`pa.large_binary()` in Python, `LargeBinary` in TypeScript, and
|
||||
`DataType::LargeBinary` in Rust with `{"lance-encoding:blob": "true"}` field
|
||||
metadata). Changing a field's metadata alone does not migrate an existing V1
|
||||
column to Blob v2.
|
||||
|
||||
#### 1\. Define a blob schema {#multimodal-1-define-a-blob-schema}
|
||||
|
||||
To use the Blob API, you must mark the column with `{"lance-encoding:blob": "true"}` metadata.
|
||||
V1 blob columns use a `large_binary` field marked with `{"lance-encoding:blob": "true"}` metadata:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{BlobApiSchema}
|
||||
{LegacyBlobApiSchema}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsBlobApiSchema}
|
||||
{TsLegacyBlobApiSchema}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsBlobApiSchema}
|
||||
{RsLegacyBlobApiSchema}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
#### 2\. Ingest large blobs {#multimodal-2-ingest-large-blobs}
|
||||
|
||||
You can then ingest data normally, and Lance will handle the optimized storage.
|
||||
Write raw bytes to a V1 blob column:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{BlobApiIngest}
|
||||
{LegacyBlobApiIngest}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsBlobApiIngest}
|
||||
{TsLegacyBlobApiIngest}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsBlobApiIngest}
|
||||
{RsLegacyBlobApiIngest}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
<Card>
|
||||
For more advanced usage, including random access and file-like reading of blobs, see the
|
||||
Lance format's [blob API documentation](https://lance.org/guide/blob/).
|
||||
</Card>
|
||||
|
||||
#### 3\. Convert blob tables to pandas {#multimodal-3-convert-blob-tables-to-pandas}
|
||||
|
||||
When you call `to_pandas()` on a local LanceDB table that contains Blob API columns, the `blob_mode` argument controls how those columns materialize. This is available in the Python SDK on local tables; remote tables raise `NotImplementedError`.
|
||||
|
||||
`blob_mode` accepts:
|
||||
|
||||
- `"lazy"` (default): returns blob columns as lazy `BlobFile` objects without eagerly materializing their payloads. Use this when you want to stream blob bytes on demand or only inspect a subset of rows. Namespace-backed local tables also use the Lance native blob-aware pandas conversion for lazy blobs; in-memory datasets fall back to the standard PyArrow `to_pandas()` path.
|
||||
- `"bytes"`: eagerly materializes each blob as `bytes`. Use this when you need the raw payload in the DataFrame, for example to decode an image or audio clip in-process.
|
||||
- `"descriptions"`: returns blob descriptors (offsets, sizes, and positions) instead of the data itself. Use this when you want to plan I/O without paying the cost of loading every blob.
|
||||
|
||||
`"bytes"` is not supported on in-memory tables.
|
||||
|
||||
Extra keyword arguments are forwarded to the underlying PyArrow / Lance pandas conversion, so you can also pass options like `split_blocks` or `self_destruct`:
|
||||
On local tables with V1 blob columns, `to_pandas()` in Python accepts `blob_mode`
|
||||
(`"lazy"`, `"bytes"`, or `"descriptions"`). Remote tables raise
|
||||
`NotImplementedError`, and `"bytes"` is not supported on in-memory tables.
|
||||
Extra keyword arguments such as `split_blocks` or `self_destruct` are forwarded
|
||||
to the PyArrow / Lance pandas conversion:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{BlobApiToPandas}
|
||||
{LegacyBlobApiToPandas}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
Query builders also accept `blob_mode` on their `to_pandas()` method:
|
||||
|
||||
- Plain scans support `"lazy"`, `"bytes"`, and `"descriptions"` with filters, projections, aliases, `limit`, and `offset`.
|
||||
- Vector, FTS, hybrid, and ordered queries can't materialize blob columns through `to_pandas()`; omit blob columns from the projection for those query shapes.
|
||||
- This works on both sync and async query builders. Extra PyArrow kwargs like `split_blocks` and `self_destruct` are still forwarded.
|
||||
On query builders with V1 blob columns, plain scans support `"lazy"`, `"bytes"`,
|
||||
and `"descriptions"`. Vector, FTS, hybrid, and ordered queries cannot
|
||||
materialize V1 blob columns through `to_pandas()`, so omit those columns from
|
||||
the projection:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{QueryToPandasKwargs}
|
||||
{LegacyQueryToPandasKwargs}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
### Store binary data {#multimodal-store-binary-data}
|
||||
|
||||
Use a `binary` column when the query should return the raw bytes. Declare it as `pa.binary()` in Python, `Binary` in TypeScript, and `DataType::Binary` in Rust.
|
||||
|
||||
#### 1\. Setup and imports {#multimodal-1-setup-and-imports}
|
||||
|
||||
Import LanceDB and Arrow in your SDK.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{MultimodalImports}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsMultimodalImports}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsMultimodalImports}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
#### 2\. Prepare data {#multimodal-2-prepare-data}
|
||||
|
||||
Encode each image or file into raw bytes before writing.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{CreateDummyData}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsCreateDummyData}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsCreateDummyData}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
#### 3\. Define the schema {#multimodal-3-define-the-schema}
|
||||
|
||||
Declare the schema explicitly so Arrow and LanceDB store the field as `binary` instead of inferring a string or integer list.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{DefineSchema}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsDefineSchema}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsDefineSchema}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
#### 4\. Ingest data {#multimodal-4-ingest-data}
|
||||
|
||||
Create the table from the rows and schema.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{IngestData}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsIngestData}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsIngestData}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
### Retrieve binary data {#multimodal-retrieve-and-use-blobs}
|
||||
|
||||
When you search the table, select the `binary` column alongside your metadata columns.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{SearchData}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsSearchData}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsSearchData}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
#### Convert bytes back to objects {#multimodal-convert-bytes-back-to-objects}
|
||||
|
||||
Decode the returned bytes into an image or media buffer in your application.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{ProcessResults}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsProcessResults}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsProcessResults}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
### Other modalities {#multimodal-other-modalities}
|
||||
|
||||
The `pa.binary()` and `pa.large_binary()` types are universal. You can use this same pattern for other types of multimodal data:
|
||||
|
||||
- **Audio:** Read `.wav` or `.mp3` files as bytes.
|
||||
- **Video:** Store video transitions or full clips using the Blob API.
|
||||
- **PDFs/Documents:** Store the raw file content for document search.
|
||||
For PDFs, audio recordings, and other files, keep searchable fields in separate
|
||||
columns so you can filter rows before fetching blob bytes.
|
||||
|
||||
### bfloat16 {#multimodal-bfloat16}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user