mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-12 23:02:28 +00:00
docs: declare where storage, training and integration pages apply
Storage configuration from the connection is embedded only, since a deployment configures its own. Training data loading, PyTorch, embedding functions, pandas, Polars and Pydantic work with local and deployment tables, and the Hugging Face Hub integration is embedded only. A local table cannot be pickled for DataLoader workers, so the PyTorch page says to wrap it in a Permutation; a LazyFrame over a whole table, to_pandas in the LangChain example and to_arrow in the DuckDB note are local only; and query text is embedded in the client for both offerings.
This commit is contained in:
@@ -4,6 +4,13 @@ sidebarTitle: "Storage"
|
||||
description: "Choosing and configuring where LanceDB stores its data -- how you deploy an embedded database."
|
||||
icon: database
|
||||
keywords: ["storage", "s3", "gcs", "azure", "object store", "configuration", "deploy"]
|
||||
availability:
|
||||
oss: available
|
||||
enterprise: unavailable
|
||||
summary: >-
|
||||
The embedded library chooses and configures its storage backend from the
|
||||
connection URI and `storage_options`. On a deployment, storage is part of
|
||||
the deployment's own configuration instead.
|
||||
---
|
||||
|
||||
LanceDB's storage layer is built on modular, disk-first components. That design makes it flexible enough to run across local NVMe, EBS, EFS, and any object store that exposes an S3-compatible API. It also supports region-specific backends such as [Tencent COS](/storage/configuration#tencent-cos) and cache-acceleration layers such as [GooseFS](/storage/configuration#goosefs).
|
||||
|
||||
@@ -3,6 +3,13 @@ title: "Managing Embeddings"
|
||||
sidebarTitle: "Managing Embeddings"
|
||||
description: "Use the embedding API in LanceDB -- registry, functions, schemas, and multi-language SDK support."
|
||||
icon: "bars"
|
||||
availability:
|
||||
oss: available
|
||||
enterprise: available
|
||||
summary: >-
|
||||
Embedding functions run in your process, whether the table is local or on a
|
||||
deployment, which stores their configuration but never runs them. Python
|
||||
and TypeScript also embed query text; in Rust you embed queries yourself.
|
||||
---
|
||||
|
||||
import {
|
||||
@@ -30,8 +37,9 @@ the data, so vectors that are close to each other are considered similar.
|
||||
|
||||
LanceDB provides an embedding function registry in OSS as well as its Enterprise versions
|
||||
([see below](#embeddings-in-lancedb-enterprise))
|
||||
that automatically generates vector embeddings during data ingestion. Automatic query-time embedding
|
||||
generation is available in LanceDB OSS, with SDK-specific query ergonomics. The API abstracts
|
||||
that automatically generates vector embeddings during data ingestion. Query text is embedded
|
||||
automatically too, in the client, for local and Enterprise tables alike, with SDK-specific query
|
||||
ergonomics. The API abstracts
|
||||
embedding generation, allowing you to focus on your application logic.
|
||||
|
||||
## Embedding Registry {#embedding-registry}
|
||||
|
||||
@@ -2,6 +2,12 @@
|
||||
title: "Hugging Face Hub"
|
||||
sidebarTitle: "Hugging Face"
|
||||
description: "Use LanceDB directly on Lance datasets hosted on the Hugging Face Hub for multimodal search and retrieval."
|
||||
availability:
|
||||
oss: available
|
||||
enterprise: unavailable
|
||||
summary: >-
|
||||
The embedded library opens Lance datasets on the Hugging Face Hub in place
|
||||
through `hf://` URIs. A deployment does not read `hf://` datasets.
|
||||
---
|
||||
|
||||
[Hugging Face Hub](https://huggingface.co/datasets?format=format:lance&sort=trending) is a popular platform for sharing machine learning datasets, models, and other resources.
|
||||
|
||||
@@ -80,6 +80,9 @@ It returns list of ids of the added texts.
|
||||
{PyFrameworksLangchainAddTexts}
|
||||
</CodeBlock>
|
||||
|
||||
Against a deployment, `tbl.to_pandas()` raises in the synchronous Python client;
|
||||
read the rows with a query instead, such as `tbl.search().limit(100).to_pandas()`.
|
||||
|
||||
------
|
||||
|
||||
|
||||
|
||||
@@ -6,7 +6,7 @@ description: "Learn how to use the DuckDB-Lance extension to query Lance tables
|
||||
|
||||
LanceDB integrates with [DuckDB](https://duckdb.org/) through the [Lance extension](https://github.com/lance-format/lance-duckdb) for DuckDB. In this page, we'll show how LanceDB manages table lifecycle, and DuckDB provides SQL analytics (including joins) and search over those tables.
|
||||
|
||||
Note that earlier versions of LanceDB used to recommend converting Lance tables to Arrow tables via `table.to_arrow()`. Although this method is still available (because DuckDB [natively scans Arrow tables](https://duckdb.org/2021/12/03/duck-arrow)), it is no longer the recommended workflow for working with Lance tables in DuckDB. This page shows how to use the Lance extension with namespace-attached LanceDB tables, allowing you to pushdown SQL queries directly to the Lance layer.
|
||||
Note that earlier versions of LanceDB used to recommend converting Lance tables to Arrow tables via `table.to_arrow()`. Although this method is still available for local tables (because DuckDB [natively scans Arrow tables](https://duckdb.org/2021/12/03/duck-arrow)), it is no longer the recommended workflow for working with Lance tables in DuckDB. This page shows how to use the Lance extension with namespace-attached LanceDB tables, allowing you to pushdown SQL queries directly to the Lance layer.
|
||||
|
||||
|
||||
## Install {#install}
|
||||
|
||||
@@ -1,6 +1,12 @@
|
||||
---
|
||||
title: "Pandas and PyArrow"
|
||||
sidebarTitle: "Pandas & PyArrow"
|
||||
availability:
|
||||
oss: available
|
||||
enterprise: available
|
||||
summary: >-
|
||||
DataFrames go into `create_table` and come back from query results the same
|
||||
way for local and deployment tables.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -1,6 +1,13 @@
|
||||
---
|
||||
title: "Polars"
|
||||
sidebarTitle: "Polars & Arrow"
|
||||
availability:
|
||||
oss: available
|
||||
enterprise: available
|
||||
summary: >-
|
||||
Polars data goes into `create_table`, and query results come back as Polars
|
||||
DataFrames, for local and deployment tables alike; a LazyFrame over a whole
|
||||
table is local only.
|
||||
|
||||
---
|
||||
|
||||
@@ -36,7 +43,7 @@ Run vector search and keep the results as a Polars `DataFrame` for further proce
|
||||
|
||||
## Work with LazyFrames {#work-with-lazyframes}
|
||||
|
||||
When you want to operate on the entire table (potentially larger than RAM), convert to a Polars `LazyFrame` so you can chain transformations without loading everything at once.
|
||||
When you want to operate on the entire table (potentially larger than RAM), convert to a Polars `LazyFrame` so you can chain transformations without loading everything at once. This works for local tables only; against a deployment, `search(...).to_polars()` returns an eager DataFrame of query results.
|
||||
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{PyPlatformsPolarsLazyframe}
|
||||
|
||||
@@ -1,6 +1,13 @@
|
||||
---
|
||||
title: "Pydantic"
|
||||
sidebarTitle: "Pydantic"
|
||||
availability:
|
||||
oss: available
|
||||
enterprise: available
|
||||
summary: >-
|
||||
The SDK converts `LanceModel` schemas to Arrow, and query rows back to
|
||||
models, in your process, so they work the same with local and deployment
|
||||
tables.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -3,6 +3,12 @@ title: "Loading Data for Model Training"
|
||||
sidebarTitle: "Data Loading & Shuffles"
|
||||
description: "Stream, shuffle, transform, and resume model training data with LanceDB."
|
||||
icon: boxes-stacked
|
||||
availability:
|
||||
oss: available
|
||||
enterprise: available
|
||||
summary: >-
|
||||
Streaming, shuffling and resuming training data work with local,
|
||||
object-storage and deployment tables in the Python SDK.
|
||||
---
|
||||
|
||||
LanceDB provides an excellent data backend for training machine learning models. For basic use cases, a `Table` can be fed directly into a data loader. For a more complete solution, a `StreamingDataset` adds prefetching, elastic determinism, resumability, multithreaded transformations, and more.
|
||||
|
||||
@@ -3,6 +3,12 @@ title: "PyTorch Integration"
|
||||
sidebarTitle: "PyTorch"
|
||||
description: Learn how to use LanceDB with PyTorch for training and inference.
|
||||
icon: fire
|
||||
availability:
|
||||
oss: available
|
||||
enterprise: available
|
||||
summary: >-
|
||||
The PyTorch integration works with local, object-storage and deployment
|
||||
tables in the Python SDK.
|
||||
---
|
||||
|
||||
LanceDB provides a seamless integration with PyTorch for training and inference. This allows you to use LanceDB as a backend for your PyTorch models, and to use PyTorch for training and inference. You can use LanceDB to store your data, and PyTorch to train your models.
|
||||
@@ -106,7 +112,7 @@ for batch in dataloader:
|
||||
|
||||
## Using multiple DataLoader workers {#using-multiple-dataloader-workers}
|
||||
|
||||
Set `num_workers > 0` to read from LanceDB in multiple PyTorch worker processes. LanceDB tables and `Permutation` objects are picklable, so each worker reopens the table after it starts.
|
||||
Set `num_workers > 0` to read from LanceDB in multiple PyTorch worker processes. `Permutation` objects and tables on a deployment are picklable, so each worker reopens the table after it starts. A local table is not, so wrap it in a `Permutation`, as below, to read it from several workers.
|
||||
|
||||
Prefer the `forkserver` start method when using multiple workers. LanceDB uses internal threads, so the default `fork` method is unsafe; `forkserver` avoids that while being cheaper to start than `spawn`, and it is set to become the Python default. See [the performance guide](/search#performance) for more multiprocessing guidance.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user