diff --git a/docs/web/api-reference/rest/index.mdx b/docs/web/api-reference/rest/index.mdx index 1442225fc..805f73cda 100644 --- a/docs/web/api-reference/rest/index.mdx +++ b/docs/web/api-reference/rest/index.mdx @@ -46,7 +46,7 @@ export LANCEDB_DATABASE="your-database-name" 4. Check that you can reach the deployment and list tables: ```bash -curl -X GET "$LANCEDB_URI/v1/tables" \ +curl -X GET "$LANCEDB_URI/v1/management/table" \ -H "Content-Type: application/json" \ -H "x-api-key: $LANCEDB_API_KEY" \ -H "x-lancedb-database: $LANCEDB_DATABASE" @@ -66,7 +66,7 @@ curl -X POST "$LANCEDB_URI/v1/tables/words" \ 6. Check that the table has been created: ```bash -curl -X GET "$LANCEDB_URI/v1/tables" \ +curl -X GET "$LANCEDB_URI/v1/management/table" \ -H "Content-Type: application/json" \ -H "x-api-key: $LANCEDB_API_KEY" \ -H "x-lancedb-database: $LANCEDB_DATABASE" diff --git a/docs/web/docs.json b/docs/web/docs.json index 11ae0be10..3a152537d 100644 --- a/docs/web/docs.json +++ b/docs/web/docs.json @@ -73,7 +73,7 @@ }, { "source": "/tables/multimodal", - "destination": "/basics/data-types#blob" + "destination": "/management/table#multimodal" }, { "source": "/search/select", @@ -220,7 +220,6 @@ "management/database", "management/namespace", "management/table", - "tables/multimodal", "tables/branching", "management/materialized-view", "management/function" diff --git a/docs/web/index.mdx b/docs/web/index.mdx index 7bc808da7..807cff9f8 100644 --- a/docs/web/index.mdx +++ b/docs/web/index.mdx @@ -8,7 +8,7 @@ keywords: ["multimodal lakehouse", "training", "feature engineering", "search", **LanceDB** is a [multimodal lakehouse](https://lancedb.com/blog/multimodal-lakehouse/) for AI teams that need one data layer for curation, feature engineering, search and retrieval, and model training. -It is built on top of [Lance](/lance), an open-source lakehouse format designed for multimodal AI data. +It is built on top of [Lance](/basics/lance), an open-source lakehouse format designed for multimodal AI data. Move from data exploration to model training on one, unified platform without needing to manage a fragmented stack of storage, feature, retrieval, and training systems. diff --git a/docs/web/management/namespace.mdx b/docs/web/management/namespace.mdx index bbaf2d26a..a7abc90b5 100644 --- a/docs/web/management/namespace.mdx +++ b/docs/web/management/namespace.mdx @@ -159,7 +159,7 @@ println!("Created table: {}", table.name()); The example above showed local directory-based namespaces. LanceDB also supports namespaces backed by remote object stores and external catalogs, via the REST namespace implementation. -For remote object stores with central metadata/catalog services (either commercial or open source), +For remote object stores with central metadata/basics/catalog services (either commercial or open source), use the REST namespace implementation. It is backed by REST routes (for example `POST /v1/namespace/{id}/create` and `GET /v1/namespace/{id}/list`) and server-provided table locations. diff --git a/docs/web/management/table.mdx b/docs/web/management/table.mdx index a54d2cb12..fff50e03b 100644 --- a/docs/web/management/table.mdx +++ b/docs/web/management/table.mdx @@ -84,6 +84,34 @@ import { RsVersioningTags as RsVersioningTags, RsVersioningMakeQuotesReader as RsVersioningMakeQuotesReader, } from '/snippets/tables.mdx'; +import { + PyMultimodalImports as MultimodalImports, + TsMultimodalImports as TsMultimodalImports, + RsMultimodalImports as RsMultimodalImports, + PyCreateDummyData as CreateDummyData, + TsCreateDummyData as TsCreateDummyData, + RsCreateDummyData as RsCreateDummyData, + PyDefineSchema as DefineSchema, + TsDefineSchema as TsDefineSchema, + RsDefineSchema as RsDefineSchema, + PyIngestData as IngestData, + TsIngestData as TsIngestData, + RsIngestData as RsIngestData, + PySearchData as SearchData, + TsSearchData as TsSearchData, + RsSearchData as RsSearchData, + PyProcessResults as ProcessResults, + TsProcessResults as TsProcessResults, + RsProcessResults as RsProcessResults, + PyBlobApiSchema as BlobApiSchema, + TsBlobApiSchema as TsBlobApiSchema, + RsBlobApiSchema as RsBlobApiSchema, + PyBlobApiIngest as BlobApiIngest, + TsBlobApiIngest as TsBlobApiIngest, + RsBlobApiIngest as RsBlobApiIngest, + PyBlobApiToPandas as BlobApiToPandas, + PyQueryToPandasKwargs as QueryToPandasKwargs, +} from '/snippets/multimodal.mdx'; A table is rows and columns, versioned. It is what searches read, what writes commit to, and what indexes and branches belong to. @@ -233,7 +261,7 @@ as it is to the format. | String and binary | `utf8`, `large_utf8`, `binary`, `large_binary`, `fixed_size_binary(n)` | | Date and time | `date32`, `date64`, `time32`, `time64`, `timestamp`, `duration` | | Nested | `struct`, `list`, `large_list`, `fixed_size_list`, `map` | -| Lance extensions | vectors, [JSON](/sql/data-types#json), [blobs](/tables/multimodal), bfloat16 | +| Lance extensions | vectors, [JSON](/sql/data-types#json), [blobs](#multimodal), bfloat16 | A vector column is a `fixed_size_list` -- an ordinary Arrow type, which is why it is selectable and joinable but why arithmetic over it is list @@ -242,6 +270,219 @@ arithmetic rather than scalar. See [SQL data types](/sql/data-types) for the full reference, including the extension types and pseudo-columns. +## Multimodal data {#multimodal} + +Binary data -- images, audio, video, model artifacts -- lives in the table +beside the structured columns and embeddings that describe it, and is governed +by the same versioning and search. + +Small values go in an ordinary `binary` column. Large ones go in a **blob** +column, stored out of line so a scan over the other columns stays fast. See +[Data types](/sql/data-types#blobs) for how the two differ to a reader. + +### Store binary data {#multimodal-store-binary-data} + +To store binary data, define a binary Arrow field in your schema (`pa.binary()` in Python, `Binary` in TypeScript, and `DataType::Binary` in Rust). + +#### 1\. Setup and imports {#multimodal-1-setup-and-imports} + +First, import the necessary libraries for LanceDB and Arrow in your SDK. + + + + {MultimodalImports} + + + + {TsMultimodalImports} + + + + {RsMultimodalImports} + + + +#### 2\. Prepare data {#multimodal-2-prepare-data} + +For this example, we'll create some dummy in-memory images. In a real application, you would read these from files or an API. The key is to convert your data (image, audio, etc.) into a raw `bytes` object. + + + + {CreateDummyData} + + + + {TsCreateDummyData} + + + + {RsCreateDummyData} + + + +#### 3\. Define the schema {#multimodal-3-define-the-schema} + +When creating the table, it is **highly recommended** to define the schema explicitly. This ensures that your binary data is correctly interpreted as a `binary` type by Arrow/LanceDB and not as a generic string or list. + + + + {DefineSchema} + + + + {TsDefineSchema} + + + + {RsDefineSchema} + + + +#### 4\. Ingest data {#multimodal-4-ingest-data} + +Now, create the table using the data and the defined schema. + + + + {IngestData} + + + + {TsIngestData} + + + + {RsIngestData} + + + +### Retrieve and use blobs {#multimodal-retrieve-and-use-blobs} + +When you search your LanceDB table, you can retrieve the binary column just like any other metadata. + + + + {SearchData} + + + + {TsSearchData} + + + + {RsSearchData} + + + +#### Convert bytes back to objects {#multimodal-convert-bytes-back-to-objects} + +Once you have the bytes back from the search result, you can decode them into the original format (for example, an image object or audio buffer). + + + + {ProcessResults} + + + + {TsProcessResults} + + + + {RsProcessResults} + + + +### Large Blobs (Blob API) {#multimodal-large-blobs-blob-api} + +For larger files like high-resolution images or videos, Lance provides a specialized **Blob API**. By using a large-binary Arrow type (`pa.large_binary()` in Python, `LargeBinary` in TypeScript, and `DataType::LargeBinary` in Rust) and specific metadata, you enable **lazy loading** and optimized encoding. This allows you to work with massive datasets without loading all binary data into memory upfront. + +#### 1\. Define a blob schema {#multimodal-1-define-a-blob-schema} + +To use the Blob API, you must mark the column with `{"lance-encoding:blob": "true"}` metadata. + + + + {BlobApiSchema} + + + + {TsBlobApiSchema} + + + + {RsBlobApiSchema} + + + +#### 2\. Ingest large blobs {#multimodal-2-ingest-large-blobs} + +You can then ingest data normally, and Lance will handle the optimized storage. + + + + {BlobApiIngest} + + + + {TsBlobApiIngest} + + + + {RsBlobApiIngest} + + + + +For more advanced usage, including random access and file-like reading of blobs, see the +Lance format's [blob API documentation](https://lance.org/guide/blob/). + + +#### 3\. Convert blob tables to pandas {#multimodal-3-convert-blob-tables-to-pandas} + +When you call `to_pandas()` on a local LanceDB table that contains Blob API columns, the `blob_mode` argument controls how those columns materialize. This is available in the Python SDK on local tables; remote tables raise `NotImplementedError`. + +`blob_mode` accepts: + +- `"lazy"` (default): returns blob columns as lazy `BlobFile` objects without eagerly materializing their payloads. Use this when you want to stream blob bytes on demand or only inspect a subset of rows. Namespace-backed local tables also use the Lance native blob-aware pandas conversion for lazy blobs; in-memory datasets fall back to the standard PyArrow `to_pandas()` path. +- `"bytes"`: eagerly materializes each blob as `bytes`. Use this when you need the raw payload in the DataFrame, for example to decode an image or audio clip in-process. +- `"descriptions"`: returns blob descriptors (offsets, sizes, and positions) instead of the data itself. Use this when you want to plan I/O without paying the cost of loading every blob. + +`"bytes"` and `"descriptions"` require a filesystem-backed Lance dataset and are not supported on in-memory tables. + +Extra keyword arguments are forwarded to the underlying PyArrow / Lance pandas conversion, so you can also pass options like `split_blocks` or `self_destruct`: + + + + {BlobApiToPandas} + + + +Query builders also accept `blob_mode` on their `to_pandas()` method: + +- Plain scans support `"lazy"`, `"bytes"`, and `"descriptions"` with filters, projections, aliases, `limit`, and `offset`. +- Vector, FTS, hybrid, and ordered queries can't materialize blob columns through `to_pandas()`; omit blob columns from the projection for those query shapes. +- This works on both sync and async query builders. Extra PyArrow kwargs like `split_blocks` and `self_destruct` are still forwarded. + + + + {QueryToPandasKwargs} + + + +### Other modalities {#multimodal-other-modalities} + +The `pa.binary()` and `pa.large_binary()` types are universal. You can use this same pattern for other types of multimodal data: + +- **Audio:** Read `.wav` or `.mp3` files as bytes. +- **Video:** Store video transitions or full clips using the Blob API. +- **PDFs/Documents:** Store the raw file content for document search. + +### bfloat16 {#multimodal-bfloat16} + +Sixteen-bit floats in the bfloat16 layout, stored as `fixed_size_binary(2)`. +Useful as a vector element type when halving the memory matters more than the +precision does. + ## Schema evolution {#schema} Schema evolution enables non-breaking modifications to a database table's structure — such as adding columns, altering data types, or dropping fields — to adapt to evolving data requirements without service interruptions. diff --git a/docs/web/quickstart.mdx b/docs/web/quickstart.mdx index c010ed8fb..2e38402e3 100644 --- a/docs/web/quickstart.mdx +++ b/docs/web/quickstart.mdx @@ -454,7 +454,7 @@ These snippets load the local image file and store the bytes in an `image` colum -For more examples, see [Data types](/basics/data-types#blob). +For more examples, see [Data types](/management/table#multimodal). ## 8\. Train on the result {#8-train} diff --git a/docs/web/search/index.mdx b/docs/web/search/index.mdx index 72df0c47b..4197acc78 100644 --- a/docs/web/search/index.mdx +++ b/docs/web/search/index.mdx @@ -8,7 +8,7 @@ icon: "list" | Feature | Description | |:---------------|:------------| | [Vector Search](/search/vector-search/) | Semantic similarity search with multiple distance metrics | -| [Multivector Search](/search/multivector-search/) | Search using multiple vector embeddings per document | +| [Multivector Search](/search/vector-search) | Search using multiple vector embeddings per document | | [Full-Text Search](/search/full-text-search/) | Keyword-based search with BM25 and pre-filtering | | [Hybrid Search](/search/hybrid-search/) | Combines vector and full-text search with reranking | | [Filtering](/search/filtering/) | Filter results based on metadata fields | diff --git a/docs/web/search/vector-search.mdx b/docs/web/search/vector-search.mdx index 42efbf7bf..619163841 100644 --- a/docs/web/search/vector-search.mdx +++ b/docs/web/search/vector-search.mdx @@ -281,7 +281,7 @@ Every query vector must match the inner dimension of the multivector column; Lan Here you can see how to take 2 query vectors and find the best matching pairs between them and document vectors using late interaction. The `np.random.random(size=(2, 256))` creates a 2×256 array with two random query vectors, `.limit(5)` returns the top 5 best document-query combinations, and `.to_pandas()` provides results in a DataFrame format. -**Read more:** [Multivector search](/search/multivector-search/) +**Read more:** [Multivector search](/search/vector-search) ## Advanced Search Scenarios {#advanced-search-scenarios} diff --git a/docs/web/tables/consistency.mdx b/docs/web/tables/consistency.mdx index b4fae5281..b2cde2ce1 100644 --- a/docs/web/tables/consistency.mdx +++ b/docs/web/tables/consistency.mdx @@ -30,7 +30,7 @@ For example, a real-time dashboard might require strong consistency, while a bat fine with eventual consistency. Stronger consistency is not free — the smaller the interval, the more often each read pays the cost of refreshing against object storage, which raises per-read latency and cost. -This setting works for both local ([LanceTable](/tables-and-namespaces#understanding-tables)) and remote +This setting works for both local ([LanceTable](/management/database#uris)) and remote tables. It only affects read operations — write operations are always consistent. diff --git a/docs/web/tables/multimodal.mdx b/docs/web/tables/multimodal.mdx deleted file mode 100644 index 214505fa2..000000000 --- a/docs/web/tables/multimodal.mdx +++ /dev/null @@ -1,247 +0,0 @@ ---- -title: "Multimodal data" -sidebarTitle: "Multimodal Data" -description: "Storing images, audio and video in a table, as binary columns or blobs." -icon: photo-film -keywords: ["multimodal", "blob", "binary", "images", "video", "blob api"] ---- - -import { - PyMultimodalImports as MultimodalImports, - TsMultimodalImports as TsMultimodalImports, - RsMultimodalImports as RsMultimodalImports, - PyCreateDummyData as CreateDummyData, - TsCreateDummyData as TsCreateDummyData, - RsCreateDummyData as RsCreateDummyData, - PyDefineSchema as DefineSchema, - TsDefineSchema as TsDefineSchema, - RsDefineSchema as RsDefineSchema, - PyIngestData as IngestData, - TsIngestData as TsIngestData, - RsIngestData as RsIngestData, - PySearchData as SearchData, - TsSearchData as TsSearchData, - RsSearchData as RsSearchData, - PyProcessResults as ProcessResults, - TsProcessResults as TsProcessResults, - RsProcessResults as RsProcessResults, - PyBlobApiSchema as BlobApiSchema, - TsBlobApiSchema as TsBlobApiSchema, - RsBlobApiSchema as RsBlobApiSchema, - PyBlobApiIngest as BlobApiIngest, - TsBlobApiIngest as TsBlobApiIngest, - RsBlobApiIngest as RsBlobApiIngest, - PyBlobApiToPandas as BlobApiToPandas, - PyQueryToPandasKwargs as QueryToPandasKwargs, -} from '/snippets/multimodal.mdx'; - -Binary data -- images, audio, video, model artifacts -- lives in the table -beside the structured columns and embeddings that describe it, and is governed -by the same versioning and search. - -Small values go in an ordinary `binary` column. Large ones go in a **blob** -column, stored out of line so a scan over the other columns stays fast. See -[Data types](/sql/data-types#blobs) for how the two differ to a reader. - -## Store binary data {#store-binary-data} - -To store binary data, define a binary Arrow field in your schema (`pa.binary()` in Python, `Binary` in TypeScript, and `DataType::Binary` in Rust). - -### 1\. Setup and imports {#1-setup-and-imports} - -First, import the necessary libraries for LanceDB and Arrow in your SDK. - - - - {MultimodalImports} - - - - {TsMultimodalImports} - - - - {RsMultimodalImports} - - - -### 2\. Prepare data {#2-prepare-data} - -For this example, we'll create some dummy in-memory images. In a real application, you would read these from files or an API. The key is to convert your data (image, audio, etc.) into a raw `bytes` object. - - - - {CreateDummyData} - - - - {TsCreateDummyData} - - - - {RsCreateDummyData} - - - -### 3\. Define the schema {#3-define-the-schema} - -When creating the table, it is **highly recommended** to define the schema explicitly. This ensures that your binary data is correctly interpreted as a `binary` type by Arrow/LanceDB and not as a generic string or list. - - - - {DefineSchema} - - - - {TsDefineSchema} - - - - {RsDefineSchema} - - - -### 4\. Ingest data {#4-ingest-data} - -Now, create the table using the data and the defined schema. - - - - {IngestData} - - - - {TsIngestData} - - - - {RsIngestData} - - - -## Retrieve and use blobs {#retrieve-and-use-blobs} - -When you search your LanceDB table, you can retrieve the binary column just like any other metadata. - - - - {SearchData} - - - - {TsSearchData} - - - - {RsSearchData} - - - -### Convert bytes back to objects {#convert-bytes-back-to-objects} - -Once you have the bytes back from the search result, you can decode them into the original format (for example, an image object or audio buffer). - - - - {ProcessResults} - - - - {TsProcessResults} - - - - {RsProcessResults} - - - -## Large Blobs (Blob API) {#large-blobs-blob-api} - -For larger files like high-resolution images or videos, Lance provides a specialized **Blob API**. By using a large-binary Arrow type (`pa.large_binary()` in Python, `LargeBinary` in TypeScript, and `DataType::LargeBinary` in Rust) and specific metadata, you enable **lazy loading** and optimized encoding. This allows you to work with massive datasets without loading all binary data into memory upfront. - -### 1\. Define a blob schema {#1-define-a-blob-schema} - -To use the Blob API, you must mark the column with `{"lance-encoding:blob": "true"}` metadata. - - - - {BlobApiSchema} - - - - {TsBlobApiSchema} - - - - {RsBlobApiSchema} - - - -### 2\. Ingest large blobs {#2-ingest-large-blobs} - -You can then ingest data normally, and Lance will handle the optimized storage. - - - - {BlobApiIngest} - - - - {TsBlobApiIngest} - - - - {RsBlobApiIngest} - - - - -For more advanced usage, including random access and file-like reading of blobs, see the -Lance format's [blob API documentation](https://lance.org/guide/blob/). - - -### 3\. Convert blob tables to pandas {#3-convert-blob-tables-to-pandas} - -When you call `to_pandas()` on a local LanceDB table that contains Blob API columns, the `blob_mode` argument controls how those columns materialize. This is available in the Python SDK on local tables; remote tables raise `NotImplementedError`. - -`blob_mode` accepts: - -- `"lazy"` (default): returns blob columns as lazy `BlobFile` objects without eagerly materializing their payloads. Use this when you want to stream blob bytes on demand or only inspect a subset of rows. Namespace-backed local tables also use the Lance native blob-aware pandas conversion for lazy blobs; in-memory datasets fall back to the standard PyArrow `to_pandas()` path. -- `"bytes"`: eagerly materializes each blob as `bytes`. Use this when you need the raw payload in the DataFrame, for example to decode an image or audio clip in-process. -- `"descriptions"`: returns blob descriptors (offsets, sizes, and positions) instead of the data itself. Use this when you want to plan I/O without paying the cost of loading every blob. - -`"bytes"` and `"descriptions"` require a filesystem-backed Lance dataset and are not supported on in-memory tables. - -Extra keyword arguments are forwarded to the underlying PyArrow / Lance pandas conversion, so you can also pass options like `split_blocks` or `self_destruct`: - - - - {BlobApiToPandas} - - - -Query builders also accept `blob_mode` on their `to_pandas()` method: - -- Plain scans support `"lazy"`, `"bytes"`, and `"descriptions"` with filters, projections, aliases, `limit`, and `offset`. -- Vector, FTS, hybrid, and ordered queries can't materialize blob columns through `to_pandas()`; omit blob columns from the projection for those query shapes. -- This works on both sync and async query builders. Extra PyArrow kwargs like `split_blocks` and `self_destruct` are still forwarded. - - - - {QueryToPandasKwargs} - - - -## Other modalities {#other-modalities} - -The `pa.binary()` and `pa.large_binary()` types are universal. You can use this same pattern for other types of multimodal data: - -- **Audio:** Read `.wav` or `.mp3` files as bytes. -- **Video:** Store video transitions or full clips using the Blob API. -- **PDFs/Documents:** Store the raw file content for document search. - -## bfloat16 {#bfloat16} - -Sixteen-bit floats in the bfloat16 layout, stored as `fixed_size_binary(2)`. -Useful as a vector element type when halving the memory matters more than the -precision does.