From 0913e2a436a9790f64e8bd67dada76f2cebeaf01 Mon Sep 17 00:00:00 2001 From: Jack Ye Date: Mon, 14 Sep 2026 01:29:54 -0700 Subject: [PATCH] docs: fold Multimodal Data into the Table page, and repair stale links Multimodal Data becomes a section of Table, between Data types -- which introduces blobs -- and Schema evolution. The rest is link repair. Earlier merges left links pointing at pages that had moved, which the redirect table covered but the link checker does not follow, so they were broken in the rendered site: /catalog, /lance, /namespaces, /tables-and-namespaces, /search/multivector-search, /search/explain-analyze and /basics/data-types#blob. The redirect table itself had two conflicting entries for /tables/multimodal and four pointing at reranking pages that no longer exist; both are fixed. --- docs/web/api-reference/rest/index.mdx | 4 +- docs/web/docs.json | 3 +- docs/web/index.mdx | 2 +- docs/web/management/namespace.mdx | 2 +- docs/web/management/table.mdx | 243 ++++++++++++++++++++++++- docs/web/quickstart.mdx | 2 +- docs/web/search/index.mdx | 2 +- docs/web/search/vector-search.mdx | 2 +- docs/web/tables/consistency.mdx | 2 +- docs/web/tables/multimodal.mdx | 247 -------------------------- 10 files changed, 251 insertions(+), 258 deletions(-) delete mode 100644 docs/web/tables/multimodal.mdx diff --git a/docs/web/api-reference/rest/index.mdx b/docs/web/api-reference/rest/index.mdx index 1442225fc..805f73cda 100644 --- a/docs/web/api-reference/rest/index.mdx +++ b/docs/web/api-reference/rest/index.mdx @@ -46,7 +46,7 @@ export LANCEDB_DATABASE="your-database-name" 4. Check that you can reach the deployment and list tables: ```bash -curl -X GET "$LANCEDB_URI/v1/tables" \ +curl -X GET "$LANCEDB_URI/v1/management/table" \ -H "Content-Type: application/json" \ -H "x-api-key: $LANCEDB_API_KEY" \ -H "x-lancedb-database: $LANCEDB_DATABASE" @@ -66,7 +66,7 @@ curl -X POST "$LANCEDB_URI/v1/tables/words" \ 6. Check that the table has been created: ```bash -curl -X GET "$LANCEDB_URI/v1/tables" \ +curl -X GET "$LANCEDB_URI/v1/management/table" \ -H "Content-Type: application/json" \ -H "x-api-key: $LANCEDB_API_KEY" \ -H "x-lancedb-database: $LANCEDB_DATABASE" diff --git a/docs/web/docs.json b/docs/web/docs.json index 11ae0be10..3a152537d 100644 --- a/docs/web/docs.json +++ b/docs/web/docs.json @@ -73,7 +73,7 @@ }, { "source": "/tables/multimodal", - "destination": "/basics/data-types#blob" + "destination": "/management/table#multimodal" }, { "source": "/search/select", @@ -220,7 +220,6 @@ "management/database", "management/namespace", "management/table", - "tables/multimodal", "tables/branching", "management/materialized-view", "management/function" diff --git a/docs/web/index.mdx b/docs/web/index.mdx index 7bc808da7..807cff9f8 100644 --- a/docs/web/index.mdx +++ b/docs/web/index.mdx @@ -8,7 +8,7 @@ keywords: ["multimodal lakehouse", "training", "feature engineering", "search", **LanceDB** is a [multimodal lakehouse](https://lancedb.com/blog/multimodal-lakehouse/) for AI teams that need one data layer for curation, feature engineering, search and retrieval, and model training. -It is built on top of [Lance](/lance), an open-source lakehouse format designed for multimodal AI data. +It is built on top of [Lance](/basics/lance), an open-source lakehouse format designed for multimodal AI data. Move from data exploration to model training on one, unified platform without needing to manage a fragmented stack of storage, feature, retrieval, and training systems. diff --git a/docs/web/management/namespace.mdx b/docs/web/management/namespace.mdx index bbaf2d26a..a7abc90b5 100644 --- a/docs/web/management/namespace.mdx +++ b/docs/web/management/namespace.mdx @@ -159,7 +159,7 @@ println!("Created table: {}", table.name()); The example above showed local directory-based namespaces. LanceDB also supports namespaces backed by remote object stores and external catalogs, via the REST namespace implementation. -For remote object stores with central metadata/catalog services (either commercial or open source), +For remote object stores with central metadata/basics/catalog services (either commercial or open source), use the REST namespace implementation. It is backed by REST routes (for example `POST /v1/namespace/{id}/create` and `GET /v1/namespace/{id}/list`) and server-provided table locations. diff --git a/docs/web/management/table.mdx b/docs/web/management/table.mdx index a54d2cb12..fff50e03b 100644 --- a/docs/web/management/table.mdx +++ b/docs/web/management/table.mdx @@ -84,6 +84,34 @@ import { RsVersioningTags as RsVersioningTags, RsVersioningMakeQuotesReader as RsVersioningMakeQuotesReader, } from '/snippets/tables.mdx'; +import { + PyMultimodalImports as MultimodalImports, + TsMultimodalImports as TsMultimodalImports, + RsMultimodalImports as RsMultimodalImports, + PyCreateDummyData as CreateDummyData, + TsCreateDummyData as TsCreateDummyData, + RsCreateDummyData as RsCreateDummyData, + PyDefineSchema as DefineSchema, + TsDefineSchema as TsDefineSchema, + RsDefineSchema as RsDefineSchema, + PyIngestData as IngestData, + TsIngestData as TsIngestData, + RsIngestData as RsIngestData, + PySearchData as SearchData, + TsSearchData as TsSearchData, + RsSearchData as RsSearchData, + PyProcessResults as ProcessResults, + TsProcessResults as TsProcessResults, + RsProcessResults as RsProcessResults, + PyBlobApiSchema as BlobApiSchema, + TsBlobApiSchema as TsBlobApiSchema, + RsBlobApiSchema as RsBlobApiSchema, + PyBlobApiIngest as BlobApiIngest, + TsBlobApiIngest as TsBlobApiIngest, + RsBlobApiIngest as RsBlobApiIngest, + PyBlobApiToPandas as BlobApiToPandas, + PyQueryToPandasKwargs as QueryToPandasKwargs, +} from '/snippets/multimodal.mdx'; A table is rows and columns, versioned. It is what searches read, what writes commit to, and what indexes and branches belong to. @@ -233,7 +261,7 @@ as it is to the format. | String and binary | `utf8`, `large_utf8`, `binary`, `large_binary`, `fixed_size_binary(n)` | | Date and time | `date32`, `date64`, `time32`, `time64`, `timestamp`, `duration` | | Nested | `struct`, `list`, `large_list`, `fixed_size_list`, `map` | -| Lance extensions | vectors, [JSON](/sql/data-types#json), [blobs](/tables/multimodal), bfloat16 | +| Lance extensions | vectors, [JSON](/sql/data-types#json), [blobs](#multimodal), bfloat16 | A vector column is a `fixed_size_list` -- an ordinary Arrow type, which is why it is selectable and joinable but why arithmetic over it is list @@ -242,6 +270,219 @@ arithmetic rather than scalar. See [SQL data types](/sql/data-types) for the full reference, including the extension types and pseudo-columns. +## Multimodal data {#multimodal} + +Binary data -- images, audio, video, model artifacts -- lives in the table +beside the structured columns and embeddings that describe it, and is governed +by the same versioning and search. + +Small values go in an ordinary `binary` column. Large ones go in a **blob** +column, stored out of line so a scan over the other columns stays fast. See +[Data types](/sql/data-types#blobs) for how the two differ to a reader. + +### Store binary data {#multimodal-store-binary-data} + +To store binary data, define a binary Arrow field in your schema (`pa.binary()` in Python, `Binary` in TypeScript, and `DataType::Binary` in Rust). + +#### 1\. Setup and imports {#multimodal-1-setup-and-imports} + +First, import the necessary libraries for LanceDB and Arrow in your SDK. + + + + {MultimodalImports} + + + + {TsMultimodalImports} + + + + {RsMultimodalImports} + + + +#### 2\. Prepare data {#multimodal-2-prepare-data} + +For this example, we'll create some dummy in-memory images. In a real application, you would read these from files or an API. The key is to convert your data (image, audio, etc.) into a raw `bytes` object. + + + + {CreateDummyData} + + + + {TsCreateDummyData} + + + + {RsCreateDummyData} + + + +#### 3\. Define the schema {#multimodal-3-define-the-schema} + +When creating the table, it is **highly recommended** to define the schema explicitly. This ensures that your binary data is correctly interpreted as a `binary` type by Arrow/LanceDB and not as a generic string or list. + + + + {DefineSchema} + + + + {TsDefineSchema} + + + + {RsDefineSchema} + + + +#### 4\. Ingest data {#multimodal-4-ingest-data} + +Now, create the table using the data and the defined schema. + + + + {IngestData} + + + + {TsIngestData} + + + + {RsIngestData} + + + +### Retrieve and use blobs {#multimodal-retrieve-and-use-blobs} + +When you search your LanceDB table, you can retrieve the binary column just like any other metadata. + + + + {SearchData} + + + + {TsSearchData} + + + + {RsSearchData} + + + +#### Convert bytes back to objects {#multimodal-convert-bytes-back-to-objects} + +Once you have the bytes back from the search result, you can decode them into the original format (for example, an image object or audio buffer). + + + + {ProcessResults} + + + + {TsProcessResults} + + + + {RsProcessResults} + + + +### Large Blobs (Blob API) {#multimodal-large-blobs-blob-api} + +For larger files like high-resolution images or videos, Lance provides a specialized **Blob API**. By using a large-binary Arrow type (`pa.large_binary()` in Python, `LargeBinary` in TypeScript, and `DataType::LargeBinary` in Rust) and specific metadata, you enable **lazy loading** and optimized encoding. This allows you to work with massive datasets without loading all binary data into memory upfront. + +#### 1\. Define a blob schema {#multimodal-1-define-a-blob-schema} + +To use the Blob API, you must mark the column with `{"lance-encoding:blob": "true"}` metadata. + + + + {BlobApiSchema} + + + + {TsBlobApiSchema} + + + + {RsBlobApiSchema} + + + +#### 2\. Ingest large blobs {#multimodal-2-ingest-large-blobs} + +You can then ingest data normally, and Lance will handle the optimized storage. + + + + {BlobApiIngest} + + + + {TsBlobApiIngest} + + + + {RsBlobApiIngest} + + + + +For more advanced usage, including random access and file-like reading of blobs, see the +Lance format's [blob API documentation](https://lance.org/guide/blob/). + + +#### 3\. Convert blob tables to pandas {#multimodal-3-convert-blob-tables-to-pandas} + +When you call `to_pandas()` on a local LanceDB table that contains Blob API columns, the `blob_mode` argument controls how those columns materialize. This is available in the Python SDK on local tables; remote tables raise `NotImplementedError`. + +`blob_mode` accepts: + +- `"lazy"` (default): returns blob columns as lazy `BlobFile` objects without eagerly materializing their payloads. Use this when you want to stream blob bytes on demand or only inspect a subset of rows. Namespace-backed local tables also use the Lance native blob-aware pandas conversion for lazy blobs; in-memory datasets fall back to the standard PyArrow `to_pandas()` path. +- `"bytes"`: eagerly materializes each blob as `bytes`. Use this when you need the raw payload in the DataFrame, for example to decode an image or audio clip in-process. +- `"descriptions"`: returns blob descriptors (offsets, sizes, and positions) instead of the data itself. Use this when you want to plan I/O without paying the cost of loading every blob. + +`"bytes"` and `"descriptions"` require a filesystem-backed Lance dataset and are not supported on in-memory tables. + +Extra keyword arguments are forwarded to the underlying PyArrow / Lance pandas conversion, so you can also pass options like `split_blocks` or `self_destruct`: + + + + {BlobApiToPandas} + + + +Query builders also accept `blob_mode` on their `to_pandas()` method: + +- Plain scans support `"lazy"`, `"bytes"`, and `"descriptions"` with filters, projections, aliases, `limit`, and `offset`. +- Vector, FTS, hybrid, and ordered queries can't materialize blob columns through `to_pandas()`; omit blob columns from the projection for those query shapes. +- This works on both sync and async query builders. Extra PyArrow kwargs like `split_blocks` and `self_destruct` are still forwarded. + + + + {QueryToPandasKwargs} + + + +### Other modalities {#multimodal-other-modalities} + +The `pa.binary()` and `pa.large_binary()` types are universal. You can use this same pattern for other types of multimodal data: + +- **Audio:** Read `.wav` or `.mp3` files as bytes. +- **Video:** Store video transitions or full clips using the Blob API. +- **PDFs/Documents:** Store the raw file content for document search. + +### bfloat16 {#multimodal-bfloat16} + +Sixteen-bit floats in the bfloat16 layout, stored as `fixed_size_binary(2)`. +Useful as a vector element type when halving the memory matters more than the +precision does. + ## Schema evolution {#schema} Schema evolution enables non-breaking modifications to a database table's structure — such as adding columns, altering data types, or dropping fields — to adapt to evolving data requirements without service interruptions. diff --git a/docs/web/quickstart.mdx b/docs/web/quickstart.mdx index c010ed8fb..2e38402e3 100644 --- a/docs/web/quickstart.mdx +++ b/docs/web/quickstart.mdx @@ -454,7 +454,7 @@ These snippets load the local image file and store the bytes in an `image` colum -For more examples, see [Data types](/basics/data-types#blob). +For more examples, see [Data types](/management/table#multimodal). ## 8\. Train on the result {#8-train} diff --git a/docs/web/search/index.mdx b/docs/web/search/index.mdx index 72df0c47b..4197acc78 100644 --- a/docs/web/search/index.mdx +++ b/docs/web/search/index.mdx @@ -8,7 +8,7 @@ icon: "list" | Feature | Description | |:---------------|:------------| | [Vector Search](/search/vector-search/) | Semantic similarity search with multiple distance metrics | -| [Multivector Search](/search/multivector-search/) | Search using multiple vector embeddings per document | +| [Multivector Search](/search/vector-search) | Search using multiple vector embeddings per document | | [Full-Text Search](/search/full-text-search/) | Keyword-based search with BM25 and pre-filtering | | [Hybrid Search](/search/hybrid-search/) | Combines vector and full-text search with reranking | | [Filtering](/search/filtering/) | Filter results based on metadata fields | diff --git a/docs/web/search/vector-search.mdx b/docs/web/search/vector-search.mdx index 42efbf7bf..619163841 100644 --- a/docs/web/search/vector-search.mdx +++ b/docs/web/search/vector-search.mdx @@ -281,7 +281,7 @@ Every query vector must match the inner dimension of the multivector column; Lan Here you can see how to take 2 query vectors and find the best matching pairs between them and document vectors using late interaction. The `np.random.random(size=(2, 256))` creates a 2×256 array with two random query vectors, `.limit(5)` returns the top 5 best document-query combinations, and `.to_pandas()` provides results in a DataFrame format. -**Read more:** [Multivector search](/search/multivector-search/) +**Read more:** [Multivector search](/search/vector-search) ## Advanced Search Scenarios {#advanced-search-scenarios} diff --git a/docs/web/tables/consistency.mdx b/docs/web/tables/consistency.mdx index b4fae5281..b2cde2ce1 100644 --- a/docs/web/tables/consistency.mdx +++ b/docs/web/tables/consistency.mdx @@ -30,7 +30,7 @@ For example, a real-time dashboard might require strong consistency, while a bat fine with eventual consistency. Stronger consistency is not free — the smaller the interval, the more often each read pays the cost of refreshing against object storage, which raises per-read latency and cost. -This setting works for both local ([LanceTable](/tables-and-namespaces#understanding-tables)) and remote +This setting works for both local ([LanceTable](/management/database#uris)) and remote tables. It only affects read operations — write operations are always consistent. diff --git a/docs/web/tables/multimodal.mdx b/docs/web/tables/multimodal.mdx deleted file mode 100644 index 214505fa2..000000000 --- a/docs/web/tables/multimodal.mdx +++ /dev/null @@ -1,247 +0,0 @@ ---- -title: "Multimodal data" -sidebarTitle: "Multimodal Data" -description: "Storing images, audio and video in a table, as binary columns or blobs." -icon: photo-film -keywords: ["multimodal", "blob", "binary", "images", "video", "blob api"] ---- - -import { - PyMultimodalImports as MultimodalImports, - TsMultimodalImports as TsMultimodalImports, - RsMultimodalImports as RsMultimodalImports, - PyCreateDummyData as CreateDummyData, - TsCreateDummyData as TsCreateDummyData, - RsCreateDummyData as RsCreateDummyData, - PyDefineSchema as DefineSchema, - TsDefineSchema as TsDefineSchema, - RsDefineSchema as RsDefineSchema, - PyIngestData as IngestData, - TsIngestData as TsIngestData, - RsIngestData as RsIngestData, - PySearchData as SearchData, - TsSearchData as TsSearchData, - RsSearchData as RsSearchData, - PyProcessResults as ProcessResults, - TsProcessResults as TsProcessResults, - RsProcessResults as RsProcessResults, - PyBlobApiSchema as BlobApiSchema, - TsBlobApiSchema as TsBlobApiSchema, - RsBlobApiSchema as RsBlobApiSchema, - PyBlobApiIngest as BlobApiIngest, - TsBlobApiIngest as TsBlobApiIngest, - RsBlobApiIngest as RsBlobApiIngest, - PyBlobApiToPandas as BlobApiToPandas, - PyQueryToPandasKwargs as QueryToPandasKwargs, -} from '/snippets/multimodal.mdx'; - -Binary data -- images, audio, video, model artifacts -- lives in the table -beside the structured columns and embeddings that describe it, and is governed -by the same versioning and search. - -Small values go in an ordinary `binary` column. Large ones go in a **blob** -column, stored out of line so a scan over the other columns stays fast. See -[Data types](/sql/data-types#blobs) for how the two differ to a reader. - -## Store binary data {#store-binary-data} - -To store binary data, define a binary Arrow field in your schema (`pa.binary()` in Python, `Binary` in TypeScript, and `DataType::Binary` in Rust). - -### 1\. Setup and imports {#1-setup-and-imports} - -First, import the necessary libraries for LanceDB and Arrow in your SDK. - - - - {MultimodalImports} - - - - {TsMultimodalImports} - - - - {RsMultimodalImports} - - - -### 2\. Prepare data {#2-prepare-data} - -For this example, we'll create some dummy in-memory images. In a real application, you would read these from files or an API. The key is to convert your data (image, audio, etc.) into a raw `bytes` object. - - - - {CreateDummyData} - - - - {TsCreateDummyData} - - - - {RsCreateDummyData} - - - -### 3\. Define the schema {#3-define-the-schema} - -When creating the table, it is **highly recommended** to define the schema explicitly. This ensures that your binary data is correctly interpreted as a `binary` type by Arrow/LanceDB and not as a generic string or list. - - - - {DefineSchema} - - - - {TsDefineSchema} - - - - {RsDefineSchema} - - - -### 4\. Ingest data {#4-ingest-data} - -Now, create the table using the data and the defined schema. - - - - {IngestData} - - - - {TsIngestData} - - - - {RsIngestData} - - - -## Retrieve and use blobs {#retrieve-and-use-blobs} - -When you search your LanceDB table, you can retrieve the binary column just like any other metadata. - - - - {SearchData} - - - - {TsSearchData} - - - - {RsSearchData} - - - -### Convert bytes back to objects {#convert-bytes-back-to-objects} - -Once you have the bytes back from the search result, you can decode them into the original format (for example, an image object or audio buffer). - - - - {ProcessResults} - - - - {TsProcessResults} - - - - {RsProcessResults} - - - -## Large Blobs (Blob API) {#large-blobs-blob-api} - -For larger files like high-resolution images or videos, Lance provides a specialized **Blob API**. By using a large-binary Arrow type (`pa.large_binary()` in Python, `LargeBinary` in TypeScript, and `DataType::LargeBinary` in Rust) and specific metadata, you enable **lazy loading** and optimized encoding. This allows you to work with massive datasets without loading all binary data into memory upfront. - -### 1\. Define a blob schema {#1-define-a-blob-schema} - -To use the Blob API, you must mark the column with `{"lance-encoding:blob": "true"}` metadata. - - - - {BlobApiSchema} - - - - {TsBlobApiSchema} - - - - {RsBlobApiSchema} - - - -### 2\. Ingest large blobs {#2-ingest-large-blobs} - -You can then ingest data normally, and Lance will handle the optimized storage. - - - - {BlobApiIngest} - - - - {TsBlobApiIngest} - - - - {RsBlobApiIngest} - - - - -For more advanced usage, including random access and file-like reading of blobs, see the -Lance format's [blob API documentation](https://lance.org/guide/blob/). - - -### 3\. Convert blob tables to pandas {#3-convert-blob-tables-to-pandas} - -When you call `to_pandas()` on a local LanceDB table that contains Blob API columns, the `blob_mode` argument controls how those columns materialize. This is available in the Python SDK on local tables; remote tables raise `NotImplementedError`. - -`blob_mode` accepts: - -- `"lazy"` (default): returns blob columns as lazy `BlobFile` objects without eagerly materializing their payloads. Use this when you want to stream blob bytes on demand or only inspect a subset of rows. Namespace-backed local tables also use the Lance native blob-aware pandas conversion for lazy blobs; in-memory datasets fall back to the standard PyArrow `to_pandas()` path. -- `"bytes"`: eagerly materializes each blob as `bytes`. Use this when you need the raw payload in the DataFrame, for example to decode an image or audio clip in-process. -- `"descriptions"`: returns blob descriptors (offsets, sizes, and positions) instead of the data itself. Use this when you want to plan I/O without paying the cost of loading every blob. - -`"bytes"` and `"descriptions"` require a filesystem-backed Lance dataset and are not supported on in-memory tables. - -Extra keyword arguments are forwarded to the underlying PyArrow / Lance pandas conversion, so you can also pass options like `split_blocks` or `self_destruct`: - - - - {BlobApiToPandas} - - - -Query builders also accept `blob_mode` on their `to_pandas()` method: - -- Plain scans support `"lazy"`, `"bytes"`, and `"descriptions"` with filters, projections, aliases, `limit`, and `offset`. -- Vector, FTS, hybrid, and ordered queries can't materialize blob columns through `to_pandas()`; omit blob columns from the projection for those query shapes. -- This works on both sync and async query builders. Extra PyArrow kwargs like `split_blocks` and `self_destruct` are still forwarded. - - - - {QueryToPandasKwargs} - - - -## Other modalities {#other-modalities} - -The `pa.binary()` and `pa.large_binary()` types are universal. You can use this same pattern for other types of multimodal data: - -- **Audio:** Read `.wav` or `.mp3` files as bytes. -- **Video:** Store video transitions or full clips using the Blob API. -- **PDFs/Documents:** Store the raw file content for document search. - -## bfloat16 {#bfloat16} - -Sixteen-bit floats in the bfloat16 layout, stored as `fixed_size_binary(2)`. -Useful as a vector element type when halving the memory matters more than the -precision does.