mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-13 15:22:30 +00:00
Brendan Clement
98d13f08c8
Storage configuration from the connection is embedded only, since a deployment configures its own. Training data loading, PyTorch, embedding functions, pandas, Polars and Pydantic work with local and deployment tables, and the Hugging Face Hub integration is embedded only. A local table cannot be pickled for DataLoader workers, so the PyTorch page says to wrap it in a Permutation; a LazyFrame over a whole table, to_pandas in the LangChain example and to_arrow in the DuckDB note are local only; and query text is embedded in the client for both offerings.
59 lines
2.2 KiB
Plaintext
59 lines
2.2 KiB
Plaintext
---
|
|
title: "Polars"
|
|
sidebarTitle: "Polars & Arrow"
|
|
availability:
|
|
oss: available
|
|
enterprise: available
|
|
summary: >-
|
|
Polars data goes into `create_table`, and query results come back as Polars
|
|
DataFrames, for local and deployment tables alike; a LazyFrame over a whole
|
|
table is local only.
|
|
|
|
---
|
|
|
|
import {
|
|
PyPlatformsPolarsCreateTable,
|
|
PyPlatformsPolarsImports,
|
|
PyPlatformsPolarsLazyframe,
|
|
PyPlatformsPolarsPydantic,
|
|
PyPlatformsPolarsVectorSearch,
|
|
} from '/snippets/integrations.mdx';
|
|
|
|
LanceDB supports [Polars](https://github.com/pola-rs/polars), a blazingly fast DataFrame library for Python written in Rust. Under the hood, both Lance and Polars speak Arrow, so passing data back and forth stays zero-copy and ergonomic.
|
|
|
|
## Create and Query a Table {#create-and-query-a-table}
|
|
|
|
Import the required libraries, including the optional Pydantic helpers if you plan to define schemas.
|
|
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyPlatformsPolarsImports}
|
|
</CodeBlock>
|
|
|
|
Build a Polars `DataFrame`, convert it to Arrow, and use it directly when creating a LanceDB table.
|
|
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyPlatformsPolarsCreateTable}
|
|
</CodeBlock>
|
|
|
|
Run vector search and keep the results as a Polars `DataFrame` for further processing or visualization.
|
|
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyPlatformsPolarsVectorSearch}
|
|
</CodeBlock>
|
|
|
|
## Work with LazyFrames {#work-with-lazyframes}
|
|
|
|
When you want to operate on the entire table (potentially larger than RAM), convert to a Polars `LazyFrame` so you can chain transformations without loading everything at once. This works for local tables only; against a deployment, `search(...).to_polars()` returns an eager DataFrame of query results.
|
|
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyPlatformsPolarsLazyframe}
|
|
</CodeBlock>
|
|
|
|
## Define Schemas with Pydantic {#define-schemas-with-pydantic}
|
|
|
|
You can also describe your table via `LanceModel` and continue ingesting data from Polars. This is useful when multiple teams share a schema or when you want validation.
|
|
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyPlatformsPolarsPydantic}
|
|
</CodeBlock>
|