docs: declare where storage, training and integration pages apply

Storage configuration from the connection is embedded only, since a
deployment configures its own. Training data loading, PyTorch, embedding
functions, pandas, Polars and Pydantic work with local and deployment
tables, and the Hugging Face Hub integration is embedded only.

A local table cannot be pickled for DataLoader workers, so the PyTorch
page says to wrap it in a Permutation; a LazyFrame over a whole table,
to_pandas in the LangChain example and to_arrow in the DuckDB note are
local only; and query text is embedded in the client for both offerings.
This commit is contained in:
Brendan Clement
2026-10-08 21:31:42 -07:00
parent ac2db48af6
commit 98d13f08c8
10 changed files with 61 additions and 5 deletions
+7
View File
@@ -4,6 +4,13 @@ sidebarTitle: "Storage"
description: "Choosing and configuring where LanceDB stores its data -- how you deploy an embedded database."
icon: database
keywords: ["storage", "s3", "gcs", "azure", "object store", "configuration", "deploy"]
availability:
oss: available
enterprise: unavailable
summary: >-
The embedded library chooses and configures its storage backend from the
connection URI and `storage_options`. On a deployment, storage is part of
the deployment's own configuration instead.
---
LanceDB's storage layer is built on modular, disk-first components. That design makes it flexible enough to run across local NVMe, EBS, EFS, and any object store that exposes an S3-compatible API. It also supports region-specific backends such as [Tencent COS](/storage/configuration#tencent-cos) and cache-acceleration layers such as [GooseFS](/storage/configuration#goosefs).
+10 -2
View File
@@ -3,6 +3,13 @@ title: "Managing Embeddings"
sidebarTitle: "Managing Embeddings"
description: "Use the embedding API in LanceDB -- registry, functions, schemas, and multi-language SDK support."
icon: "bars"
availability:
oss: available
enterprise: available
summary: >-
Embedding functions run in your process, whether the table is local or on a
deployment, which stores their configuration but never runs them. Python
and TypeScript also embed query text; in Rust you embed queries yourself.
---
import {
@@ -30,8 +37,9 @@ the data, so vectors that are close to each other are considered similar.
LanceDB provides an embedding function registry in OSS as well as its Enterprise versions
([see below](#embeddings-in-lancedb-enterprise))
that automatically generates vector embeddings during data ingestion. Automatic query-time embedding
generation is available in LanceDB OSS, with SDK-specific query ergonomics. The API abstracts
that automatically generates vector embeddings during data ingestion. Query text is embedded
automatically too, in the client, for local and Enterprise tables alike, with SDK-specific query
ergonomics. The API abstracts
embedding generation, allowing you to focus on your application logic.
## Embedding Registry {#embedding-registry}
+6
View File
@@ -2,6 +2,12 @@
title: "Hugging Face Hub"
sidebarTitle: "Hugging Face"
description: "Use LanceDB directly on Lance datasets hosted on the Hugging Face Hub for multimodal search and retrieval."
availability:
oss: available
enterprise: unavailable
summary: >-
The embedded library opens Lance datasets on the Hugging Face Hub in place
through `hf://` URIs. A deployment does not read `hf://` datasets.
---
[Hugging Face Hub](https://huggingface.co/datasets?format=format:lance&sort=trending) is a popular platform for sharing machine learning datasets, models, and other resources.
+3
View File
@@ -80,6 +80,9 @@ It returns list of ids of the added texts.
{PyFrameworksLangchainAddTexts}
</CodeBlock>
Against a deployment, `tbl.to_pandas()` raises in the synchronous Python client;
read the rows with a query instead, such as `tbl.search().limit(100).to_pandas()`.
------
+1 -1
View File
@@ -6,7 +6,7 @@ description: "Learn how to use the DuckDB-Lance extension to query Lance tables
LanceDB integrates with [DuckDB](https://duckdb.org/) through the [Lance extension](https://github.com/lance-format/lance-duckdb) for DuckDB. In this page, we'll show how LanceDB manages table lifecycle, and DuckDB provides SQL analytics (including joins) and search over those tables.
Note that earlier versions of LanceDB used to recommend converting Lance tables to Arrow tables via `table.to_arrow()`. Although this method is still available (because DuckDB [natively scans Arrow tables](https://duckdb.org/2021/12/03/duck-arrow)), it is no longer the recommended workflow for working with Lance tables in DuckDB. This page shows how to use the Lance extension with namespace-attached LanceDB tables, allowing you to pushdown SQL queries directly to the Lance layer.
Note that earlier versions of LanceDB used to recommend converting Lance tables to Arrow tables via `table.to_arrow()`. Although this method is still available for local tables (because DuckDB [natively scans Arrow tables](https://duckdb.org/2021/12/03/duck-arrow)), it is no longer the recommended workflow for working with Lance tables in DuckDB. This page shows how to use the Lance extension with namespace-attached LanceDB tables, allowing you to pushdown SQL queries directly to the Lance layer.
## Install {#install}
@@ -1,6 +1,12 @@
---
title: "Pandas and PyArrow"
sidebarTitle: "Pandas & PyArrow"
availability:
oss: available
enterprise: available
summary: >-
DataFrames go into `create_table` and come back from query results the same
way for local and deployment tables.
---
+8 -1
View File
@@ -1,6 +1,13 @@
---
title: "Polars"
sidebarTitle: "Polars & Arrow"
availability:
oss: available
enterprise: available
summary: >-
Polars data goes into `create_table`, and query results come back as Polars
DataFrames, for local and deployment tables alike; a LazyFrame over a whole
table is local only.
---
@@ -36,7 +43,7 @@ Run vector search and keep the results as a Polars `DataFrame` for further proce
## Work with LazyFrames {#work-with-lazyframes}
When you want to operate on the entire table (potentially larger than RAM), convert to a Polars `LazyFrame` so you can chain transformations without loading everything at once.
When you want to operate on the entire table (potentially larger than RAM), convert to a Polars `LazyFrame` so you can chain transformations without loading everything at once. This works for local tables only; against a deployment, `search(...).to_polars()` returns an eager DataFrame of query results.
<CodeBlock filename="Python" language="Python" icon="python">
{PyPlatformsPolarsLazyframe}
+7
View File
@@ -1,6 +1,13 @@
---
title: "Pydantic"
sidebarTitle: "Pydantic"
availability:
oss: available
enterprise: available
summary: >-
The SDK converts `LanceModel` schemas to Arrow, and query rows back to
models, in your process, so they work the same with local and deployment
tables.
---
+6
View File
@@ -3,6 +3,12 @@ title: "Loading Data for Model Training"
sidebarTitle: "Data Loading & Shuffles"
description: "Stream, shuffle, transform, and resume model training data with LanceDB."
icon: boxes-stacked
availability:
oss: available
enterprise: available
summary: >-
Streaming, shuffling and resuming training data work with local,
object-storage and deployment tables in the Python SDK.
---
LanceDB provides an excellent data backend for training machine learning models. For basic use cases, a `Table` can be fed directly into a data loader. For a more complete solution, a `StreamingDataset` adds prefetching, elastic determinism, resumability, multithreaded transformations, and more.
+7 -1
View File
@@ -3,6 +3,12 @@ title: "PyTorch Integration"
sidebarTitle: "PyTorch"
description: Learn how to use LanceDB with PyTorch for training and inference.
icon: fire
availability:
oss: available
enterprise: available
summary: >-
The PyTorch integration works with local, object-storage and deployment
tables in the Python SDK.
---
LanceDB provides a seamless integration with PyTorch for training and inference. This allows you to use LanceDB as a backend for your PyTorch models, and to use PyTorch for training and inference. You can use LanceDB to store your data, and PyTorch to train your models.
@@ -106,7 +112,7 @@ for batch in dataloader:
## Using multiple DataLoader workers {#using-multiple-dataloader-workers}
Set `num_workers > 0` to read from LanceDB in multiple PyTorch worker processes. LanceDB tables and `Permutation` objects are picklable, so each worker reopens the table after it starts.
Set `num_workers > 0` to read from LanceDB in multiple PyTorch worker processes. `Permutation` objects and tables on a deployment are picklable, so each worker reopens the table after it starts. A local table is not, so wrap it in a `Permutation`, as below, to read it from several workers.
Prefer the `forkserver` start method when using multiple workers. LanceDB uses internal threads, so the default `fork` method is unsafe; `forkserver` avoids that while being cheaper to start than `spawn`, and it is set to become the Python default. See [the performance guide](/search#performance) for more multiprocessing guidance.