mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-12 23:02:28 +00:00
Brendan Clement
98d13f08c8
Storage configuration from the connection is embedded only, since a deployment configures its own. Training data loading, PyTorch, embedding functions, pandas, Polars and Pydantic work with local and deployment tables, and the Hugging Face Hub integration is embedded only. A local table cannot be pickled for DataLoader workers, so the PyTorch page says to wrap it in a Permutation; a LazyFrame over a whole table, to_pandas in the LangChain example and to_arrow in the DuckDB note are local only; and query text is embedded in the client for both offerings.
53 lines
1.6 KiB
Plaintext
53 lines
1.6 KiB
Plaintext
---
|
|
title: "Pandas and PyArrow"
|
|
sidebarTitle: "Pandas & PyArrow"
|
|
availability:
|
|
oss: available
|
|
enterprise: available
|
|
summary: >-
|
|
DataFrames go into `create_table` and come back from query results the same
|
|
way for local and deployment tables.
|
|
|
|
---
|
|
|
|
import {
|
|
PyPlatformsPandasAsyncExample,
|
|
PyPlatformsPandasCreateTable,
|
|
PyPlatformsPandasImports,
|
|
PyPlatformsPandasVectorSearch,
|
|
} from '/snippets/integrations.mdx';
|
|
|
|
Because Lance is built on top of [Apache Arrow](https://arrow.apache.org/),
|
|
LanceDB fits naturally into Pandas-first workflows. You can ingest a `DataFrame`,
|
|
query it with LanceDB's vector operators, and keep working in Pandas without any glue code.
|
|
|
|
## Create a dataset {#create-a-dataset}
|
|
|
|
Start by importing LanceDB alongside your usual Pandas utilities and connect to a temporary database.
|
|
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyPlatformsPandasImports}
|
|
</CodeBlock>
|
|
|
|
Use the familiar `pd.DataFrame` API to prepare your rows, then pass the entire frame to `db.create_table`.
|
|
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyPlatformsPandasCreateTable}
|
|
</CodeBlock>
|
|
|
|
## Vector search {#vector-search}
|
|
|
|
Queries can return Pandas frames as well, so you can immediately inspect the results or pipe them into downstream analytics.
|
|
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyPlatformsPandasVectorSearch}
|
|
</CodeBlock>
|
|
|
|
## Async API {#async-api}
|
|
|
|
For web services or background jobs that already rely on `asyncio`, use the asynchronous helpers to keep everything non-blocking.
|
|
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyPlatformsPandasAsyncExample}
|
|
</CodeBlock>
|