From 98d13f08c832a564af7b0e5724f5df7d5ddd8ed6 Mon Sep 17 00:00:00 2001 From: Brendan Clement Date: Thu, 8 Oct 2026 21:31:42 -0700 Subject: [PATCH] docs: declare where storage, training and integration pages apply Storage configuration from the connection is embedded only, since a deployment configures its own. Training data loading, PyTorch, embedding functions, pandas, Polars and Pydantic work with local and deployment tables, and the Hugging Face Hub integration is embedded only. A local table cannot be pickled for DataLoader workers, so the PyTorch page says to wrap it in a Permutation; a LazyFrame over a whole table, to_pandas in the LangChain example and to_arrow in the DuckDB note are local only; and query text is embedded in the client for both offerings. --- docs/web/deployment/storage.mdx | 7 +++++++ docs/web/embedding/index.mdx | 12 ++++++++++-- docs/web/integrations/ai/huggingface.mdx | 6 ++++++ docs/web/integrations/ai/langchain.mdx | 3 +++ docs/web/integrations/data/duckdb.mdx | 2 +- docs/web/integrations/data/pandas_and_pyarrow.mdx | 6 ++++++ docs/web/integrations/data/polars_arrow.mdx | 9 ++++++++- docs/web/integrations/data/pydantic.mdx | 7 +++++++ docs/web/training/data-loading.mdx | 6 ++++++ docs/web/training/torch.mdx | 8 +++++++- 10 files changed, 61 insertions(+), 5 deletions(-) diff --git a/docs/web/deployment/storage.mdx b/docs/web/deployment/storage.mdx index fd8550ea6..55426b23a 100644 --- a/docs/web/deployment/storage.mdx +++ b/docs/web/deployment/storage.mdx @@ -4,6 +4,13 @@ sidebarTitle: "Storage" description: "Choosing and configuring where LanceDB stores its data -- how you deploy an embedded database." icon: database keywords: ["storage", "s3", "gcs", "azure", "object store", "configuration", "deploy"] +availability: + oss: available + enterprise: unavailable + summary: >- + The embedded library chooses and configures its storage backend from the + connection URI and `storage_options`. On a deployment, storage is part of + the deployment's own configuration instead. --- LanceDB's storage layer is built on modular, disk-first components. That design makes it flexible enough to run across local NVMe, EBS, EFS, and any object store that exposes an S3-compatible API. It also supports region-specific backends such as [Tencent COS](/storage/configuration#tencent-cos) and cache-acceleration layers such as [GooseFS](/storage/configuration#goosefs). diff --git a/docs/web/embedding/index.mdx b/docs/web/embedding/index.mdx index e3568604b..f4d1c8d4a 100644 --- a/docs/web/embedding/index.mdx +++ b/docs/web/embedding/index.mdx @@ -3,6 +3,13 @@ title: "Managing Embeddings" sidebarTitle: "Managing Embeddings" description: "Use the embedding API in LanceDB -- registry, functions, schemas, and multi-language SDK support." icon: "bars" +availability: + oss: available + enterprise: available + summary: >- + Embedding functions run in your process, whether the table is local or on a + deployment, which stores their configuration but never runs them. Python + and TypeScript also embed query text; in Rust you embed queries yourself. --- import { @@ -30,8 +37,9 @@ the data, so vectors that are close to each other are considered similar. LanceDB provides an embedding function registry in OSS as well as its Enterprise versions ([see below](#embeddings-in-lancedb-enterprise)) -that automatically generates vector embeddings during data ingestion. Automatic query-time embedding -generation is available in LanceDB OSS, with SDK-specific query ergonomics. The API abstracts +that automatically generates vector embeddings during data ingestion. Query text is embedded +automatically too, in the client, for local and Enterprise tables alike, with SDK-specific query +ergonomics. The API abstracts embedding generation, allowing you to focus on your application logic. ## Embedding Registry {#embedding-registry} diff --git a/docs/web/integrations/ai/huggingface.mdx b/docs/web/integrations/ai/huggingface.mdx index 62ac7e8cd..0a38c208b 100644 --- a/docs/web/integrations/ai/huggingface.mdx +++ b/docs/web/integrations/ai/huggingface.mdx @@ -2,6 +2,12 @@ title: "Hugging Face Hub" sidebarTitle: "Hugging Face" description: "Use LanceDB directly on Lance datasets hosted on the Hugging Face Hub for multimodal search and retrieval." +availability: + oss: available + enterprise: unavailable + summary: >- + The embedded library opens Lance datasets on the Hugging Face Hub in place + through `hf://` URIs. A deployment does not read `hf://` datasets. --- [Hugging Face Hub](https://huggingface.co/datasets?format=format:lance&sort=trending) is a popular platform for sharing machine learning datasets, models, and other resources. diff --git a/docs/web/integrations/ai/langchain.mdx b/docs/web/integrations/ai/langchain.mdx index 363331093..f451aebcf 100644 --- a/docs/web/integrations/ai/langchain.mdx +++ b/docs/web/integrations/ai/langchain.mdx @@ -80,6 +80,9 @@ It returns list of ids of the added texts. {PyFrameworksLangchainAddTexts} +Against a deployment, `tbl.to_pandas()` raises in the synchronous Python client; +read the rows with a query instead, such as `tbl.search().limit(100).to_pandas()`. + ------ diff --git a/docs/web/integrations/data/duckdb.mdx b/docs/web/integrations/data/duckdb.mdx index 704a47c60..23c4891f9 100644 --- a/docs/web/integrations/data/duckdb.mdx +++ b/docs/web/integrations/data/duckdb.mdx @@ -6,7 +6,7 @@ description: "Learn how to use the DuckDB-Lance extension to query Lance tables LanceDB integrates with [DuckDB](https://duckdb.org/) through the [Lance extension](https://github.com/lance-format/lance-duckdb) for DuckDB. In this page, we'll show how LanceDB manages table lifecycle, and DuckDB provides SQL analytics (including joins) and search over those tables. -Note that earlier versions of LanceDB used to recommend converting Lance tables to Arrow tables via `table.to_arrow()`. Although this method is still available (because DuckDB [natively scans Arrow tables](https://duckdb.org/2021/12/03/duck-arrow)), it is no longer the recommended workflow for working with Lance tables in DuckDB. This page shows how to use the Lance extension with namespace-attached LanceDB tables, allowing you to pushdown SQL queries directly to the Lance layer. +Note that earlier versions of LanceDB used to recommend converting Lance tables to Arrow tables via `table.to_arrow()`. Although this method is still available for local tables (because DuckDB [natively scans Arrow tables](https://duckdb.org/2021/12/03/duck-arrow)), it is no longer the recommended workflow for working with Lance tables in DuckDB. This page shows how to use the Lance extension with namespace-attached LanceDB tables, allowing you to pushdown SQL queries directly to the Lance layer. ## Install {#install} diff --git a/docs/web/integrations/data/pandas_and_pyarrow.mdx b/docs/web/integrations/data/pandas_and_pyarrow.mdx index 170ccf772..c646e7f3b 100644 --- a/docs/web/integrations/data/pandas_and_pyarrow.mdx +++ b/docs/web/integrations/data/pandas_and_pyarrow.mdx @@ -1,6 +1,12 @@ --- title: "Pandas and PyArrow" sidebarTitle: "Pandas & PyArrow" +availability: + oss: available + enterprise: available + summary: >- + DataFrames go into `create_table` and come back from query results the same + way for local and deployment tables. --- diff --git a/docs/web/integrations/data/polars_arrow.mdx b/docs/web/integrations/data/polars_arrow.mdx index 4cb85737e..2770ed498 100644 --- a/docs/web/integrations/data/polars_arrow.mdx +++ b/docs/web/integrations/data/polars_arrow.mdx @@ -1,6 +1,13 @@ --- title: "Polars" sidebarTitle: "Polars & Arrow" +availability: + oss: available + enterprise: available + summary: >- + Polars data goes into `create_table`, and query results come back as Polars + DataFrames, for local and deployment tables alike; a LazyFrame over a whole + table is local only. --- @@ -36,7 +43,7 @@ Run vector search and keep the results as a Polars `DataFrame` for further proce ## Work with LazyFrames {#work-with-lazyframes} -When you want to operate on the entire table (potentially larger than RAM), convert to a Polars `LazyFrame` so you can chain transformations without loading everything at once. +When you want to operate on the entire table (potentially larger than RAM), convert to a Polars `LazyFrame` so you can chain transformations without loading everything at once. This works for local tables only; against a deployment, `search(...).to_polars()` returns an eager DataFrame of query results. {PyPlatformsPolarsLazyframe} diff --git a/docs/web/integrations/data/pydantic.mdx b/docs/web/integrations/data/pydantic.mdx index b6f80aca3..3ac660774 100644 --- a/docs/web/integrations/data/pydantic.mdx +++ b/docs/web/integrations/data/pydantic.mdx @@ -1,6 +1,13 @@ --- title: "Pydantic" sidebarTitle: "Pydantic" +availability: + oss: available + enterprise: available + summary: >- + The SDK converts `LanceModel` schemas to Arrow, and query rows back to + models, in your process, so they work the same with local and deployment + tables. --- diff --git a/docs/web/training/data-loading.mdx b/docs/web/training/data-loading.mdx index 8135e5898..8ace0c9d5 100644 --- a/docs/web/training/data-loading.mdx +++ b/docs/web/training/data-loading.mdx @@ -3,6 +3,12 @@ title: "Loading Data for Model Training" sidebarTitle: "Data Loading & Shuffles" description: "Stream, shuffle, transform, and resume model training data with LanceDB." icon: boxes-stacked +availability: + oss: available + enterprise: available + summary: >- + Streaming, shuffling and resuming training data work with local, + object-storage and deployment tables in the Python SDK. --- LanceDB provides an excellent data backend for training machine learning models. For basic use cases, a `Table` can be fed directly into a data loader. For a more complete solution, a `StreamingDataset` adds prefetching, elastic determinism, resumability, multithreaded transformations, and more. diff --git a/docs/web/training/torch.mdx b/docs/web/training/torch.mdx index 7019fa49b..73dbe987e 100644 --- a/docs/web/training/torch.mdx +++ b/docs/web/training/torch.mdx @@ -3,6 +3,12 @@ title: "PyTorch Integration" sidebarTitle: "PyTorch" description: Learn how to use LanceDB with PyTorch for training and inference. icon: fire +availability: + oss: available + enterprise: available + summary: >- + The PyTorch integration works with local, object-storage and deployment + tables in the Python SDK. --- LanceDB provides a seamless integration with PyTorch for training and inference. This allows you to use LanceDB as a backend for your PyTorch models, and to use PyTorch for training and inference. You can use LanceDB to store your data, and PyTorch to train your models. @@ -106,7 +112,7 @@ for batch in dataloader: ## Using multiple DataLoader workers {#using-multiple-dataloader-workers} -Set `num_workers > 0` to read from LanceDB in multiple PyTorch worker processes. LanceDB tables and `Permutation` objects are picklable, so each worker reopens the table after it starts. +Set `num_workers > 0` to read from LanceDB in multiple PyTorch worker processes. `Permutation` objects and tables on a deployment are picklable, so each worker reopens the table after it starts. A local table is not, so wrap it in a `Permutation`, as below, to read it from several workers. Prefer the `forkserver` start method when using multiple workers. LanceDB uses internal threads, so the default `fork` method is unsafe; `forkserver` avoids that while being cheaper to start than `spawn`, and it is set to become the Python default. See [the performance guide](/search#performance) for more multiprocessing guidance.