mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-16 16:52:34 +00:00
docs: declare where storage, training and integration pages apply
Storage configuration from the connection is embedded only, since a deployment configures its own. Training data loading, PyTorch, embedding functions, pandas, Polars and Pydantic work with local and deployment tables, and the Hugging Face Hub integration is embedded only. A local table cannot be pickled for DataLoader workers, so the PyTorch page says to wrap it in a Permutation; a LazyFrame over a whole table, to_pandas in the LangChain example and to_arrow in the DuckDB note are local only; and query text is embedded in the client for both offerings.
This commit is contained in:
@@ -6,7 +6,7 @@ description: "Learn how to use the DuckDB-Lance extension to query Lance tables
|
||||
|
||||
LanceDB integrates with [DuckDB](https://duckdb.org/) through the [Lance extension](https://github.com/lance-format/lance-duckdb) for DuckDB. In this page, we'll show how LanceDB manages table lifecycle, and DuckDB provides SQL analytics (including joins) and search over those tables.
|
||||
|
||||
Note that earlier versions of LanceDB used to recommend converting Lance tables to Arrow tables via `table.to_arrow()`. Although this method is still available (because DuckDB [natively scans Arrow tables](https://duckdb.org/2021/12/03/duck-arrow)), it is no longer the recommended workflow for working with Lance tables in DuckDB. This page shows how to use the Lance extension with namespace-attached LanceDB tables, allowing you to pushdown SQL queries directly to the Lance layer.
|
||||
Note that earlier versions of LanceDB used to recommend converting Lance tables to Arrow tables via `table.to_arrow()`. Although this method is still available for local tables (because DuckDB [natively scans Arrow tables](https://duckdb.org/2021/12/03/duck-arrow)), it is no longer the recommended workflow for working with Lance tables in DuckDB. This page shows how to use the Lance extension with namespace-attached LanceDB tables, allowing you to pushdown SQL queries directly to the Lance layer.
|
||||
|
||||
|
||||
## Install {#install}
|
||||
|
||||
@@ -1,6 +1,12 @@
|
||||
---
|
||||
title: "Pandas and PyArrow"
|
||||
sidebarTitle: "Pandas & PyArrow"
|
||||
availability:
|
||||
oss: available
|
||||
enterprise: available
|
||||
summary: >-
|
||||
DataFrames go into `create_table` and come back from query results the same
|
||||
way for local and deployment tables.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -1,6 +1,13 @@
|
||||
---
|
||||
title: "Polars"
|
||||
sidebarTitle: "Polars & Arrow"
|
||||
availability:
|
||||
oss: available
|
||||
enterprise: available
|
||||
summary: >-
|
||||
Polars data goes into `create_table`, and query results come back as Polars
|
||||
DataFrames, for local and deployment tables alike; a LazyFrame over a whole
|
||||
table is local only.
|
||||
|
||||
---
|
||||
|
||||
@@ -36,7 +43,7 @@ Run vector search and keep the results as a Polars `DataFrame` for further proce
|
||||
|
||||
## Work with LazyFrames {#work-with-lazyframes}
|
||||
|
||||
When you want to operate on the entire table (potentially larger than RAM), convert to a Polars `LazyFrame` so you can chain transformations without loading everything at once.
|
||||
When you want to operate on the entire table (potentially larger than RAM), convert to a Polars `LazyFrame` so you can chain transformations without loading everything at once. This works for local tables only; against a deployment, `search(...).to_polars()` returns an eager DataFrame of query results.
|
||||
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{PyPlatformsPolarsLazyframe}
|
||||
|
||||
@@ -1,6 +1,13 @@
|
||||
---
|
||||
title: "Pydantic"
|
||||
sidebarTitle: "Pydantic"
|
||||
availability:
|
||||
oss: available
|
||||
enterprise: available
|
||||
summary: >-
|
||||
The SDK converts `LanceModel` schemas to Arrow, and query rows back to
|
||||
models, in your process, so they work the same with local and deployment
|
||||
tables.
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in New Issue
Block a user