mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-30 15:26:52 +00:00
Jack Ye
ddb318567a
## Summary - Complete the SQL reference: `ATTACH DATABASE`, the `CHANGES` clause, and `dedupe()`/`duplicate_pairs()` (verified against current sophon source; these were recently renamed from `vector_dedup`/`vector_duplicate_pairs`). Fix the blob-column default-projection claim and add the previously undocumented `blob_descriptor()`. - Enrich the gated Functions draft (still `.mintignore`'d, pre-release): chained Functions over blob outputs, GPU-produced blobs, Secrets, multi-input/output binding, `code=[module]` packaging. - Upgrade the materialized-view and computed-column auto-refresh drafts with precise, verified catch-up semantics, and add the missing `ALTER MATERIALIZED VIEW` / `CREATE ... WITH(...)` sections (also gated — the automatic-refresh scheduler is still dev/staging-only). - Thread one running example (video frames → GPU extract/caption → dedup → auto-refreshing materialized view) through Functions, materialized views, and the LeRobotDataset integration guide instead of a separate tutorial. - Rename "LanceDB OSS"/"OSS" to "LanceDB Embedded"/"Embedded" throughout (52 files); activate the existing availability-comparison mechanism on `basics/offerings.mdx` for a full feature-by-feature comparison table.
54 lines
2.0 KiB
Plaintext
54 lines
2.0 KiB
Plaintext
---
|
|
title: "Polars"
|
|
sidebarTitle: "Polars & Arrow"
|
|
availability:
|
|
embedded: available
|
|
enterprise: available
|
|
---
|
|
|
|
import {
|
|
PyPlatformsPolarsCreateTable,
|
|
PyPlatformsPolarsImports,
|
|
PyPlatformsPolarsLazyframe,
|
|
PyPlatformsPolarsPydantic,
|
|
PyPlatformsPolarsVectorSearch,
|
|
} from '/snippets/integrations.mdx';
|
|
|
|
LanceDB supports [Polars](https://github.com/pola-rs/polars), a blazingly fast DataFrame library for Python written in Rust. Under the hood, both Lance and Polars speak Arrow, so passing data back and forth stays zero-copy and ergonomic.
|
|
|
|
## Create and Query a Table {#create-and-query-a-table}
|
|
|
|
Import the required libraries, including the optional Pydantic helpers if you plan to define schemas.
|
|
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyPlatformsPolarsImports}
|
|
</CodeBlock>
|
|
|
|
Build a Polars `DataFrame`, convert it to Arrow, and use it directly when creating a LanceDB table.
|
|
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyPlatformsPolarsCreateTable}
|
|
</CodeBlock>
|
|
|
|
Run vector search and keep the results as a Polars `DataFrame` for further processing or visualization.
|
|
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyPlatformsPolarsVectorSearch}
|
|
</CodeBlock>
|
|
|
|
## Work with LazyFrames {#work-with-lazyframes}
|
|
|
|
When you want to operate on the entire table (potentially larger than RAM), convert to a Polars `LazyFrame` so you can chain transformations without loading everything at once. This works for LanceDB Embedded tables only; with LanceDB Enterprise, `search(...).to_polars()` returns an eager DataFrame of query results.
|
|
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyPlatformsPolarsLazyframe}
|
|
</CodeBlock>
|
|
|
|
## Define Schemas with Pydantic {#define-schemas-with-pydantic}
|
|
|
|
You can also describe your table via `LanceModel` and continue ingesting data from Polars. This is useful when multiple teams share a schema or when you want validation.
|
|
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyPlatformsPolarsPydantic}
|
|
</CodeBlock>
|