mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-17 01:02:28 +00:00
Brendan Clement
758f738b8c
This PR is part of the documentation revamp. Once the full set of PRs is merged, we’ll publish the combined site to staging for review before updating docs.lancedb.com. Ingestion, search and integration examples are incomplete or inconsistent across languages, and some guidance misstates SDK behavior. This brings the guides and their example sources together. It includes complete Python, TypeScript and Rust workflows, isolated provider and reranker examples, the Superlinked corrections, and guidance on retries that can duplicate writes. Stacked on #4491. Each MDX file has one PR owner. Rebase onto main after #4491 is squash-merged. Local assembly, Mint validation and link checks pass. Example code is unchanged from the previously validated package. Publication stays held through integration.
97 lines
3.5 KiB
Plaintext
97 lines
3.5 KiB
Plaintext
---
|
|
title: "Pydantic"
|
|
sidebarTitle: "Pydantic"
|
|
availability:
|
|
oss: available
|
|
enterprise: available
|
|
summary: >-
|
|
The SDK converts `LanceModel` schemas to Arrow, and query rows back to
|
|
models, in your process, so they work the same with local and deployment
|
|
tables.
|
|
|
|
---
|
|
|
|
import {
|
|
PyFrameworksPydanticBaseExample,
|
|
PyFrameworksPydanticBaseModel,
|
|
PyFrameworksPydanticImports,
|
|
PyFrameworksPydanticSetUrl,
|
|
PyFrameworksPydanticTypeConversion,
|
|
PyFrameworksPydanticVectorField,
|
|
} from '/snippets/integrations.mdx';
|
|
|
|
[Pydantic](https://docs.pydantic.dev/latest/) is a data validation library in Python.
|
|
LanceDB integrates with Pydantic for schema inference, data ingestion, and query result casting.
|
|
Using `lancedb.pydantic.LanceModel`, users can seamlessly
|
|
integrate Pydantic with the rest of the LanceDB APIs.
|
|
|
|
First, import the necessary LanceDB and Pydantic modules:
|
|
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyFrameworksPydanticImports}
|
|
</CodeBlock>
|
|
|
|
Next, define your Pydantic model by inheriting from `LanceModel` and specifying your fields including a vector field:
|
|
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyFrameworksPydanticBaseModel}
|
|
</CodeBlock>
|
|
|
|
Set the database connection URL:
|
|
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyFrameworksPydanticSetUrl}
|
|
</CodeBlock>
|
|
|
|
Now you can create a table, add data, and perform vector search operations:
|
|
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyFrameworksPydanticBaseExample}
|
|
</CodeBlock>
|
|
|
|
|
|
## Vector Field {#vector-field}
|
|
|
|
LanceDB provides a `lancedb.pydantic.Vector` method to define a
|
|
vector Field in a Pydantic Model.
|
|
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyFrameworksPydanticVectorField}
|
|
</CodeBlock>
|
|
|
|
This example demonstrates how LanceDB automatically converts Pydantic field types to their corresponding Apache Arrow data types. The `pydantic_to_schema()` function takes a Pydantic model and generates an Arrow schema where:
|
|
- `int` fields become `pa.int64()` (64-bit integers)
|
|
- `str` fields become `pa.utf8()` (UTF-8 encoded strings)
|
|
- `Vector(768)` becomes `pa.list_(pa.float32(), 768)` (fixed-size list of 768 float32 values)
|
|
- The `False` parameter indicates that the fields are not nullable
|
|
|
|
## Type Conversion {#type-conversion}
|
|
|
|
LanceDB automatically convert Pydantic fields to
|
|
[Apache Arrow DataType](https://arrow.apache.org/docs/python/generated/pyarrow.DataType.html#pyarrow.DataType).
|
|
|
|
Current supported type conversions:
|
|
|
|
| Pydantic Field Type | PyArrow Data Type |
|
|
| ------------------- | ----------------- |
|
|
| `int` | `pyarrow.int64` |
|
|
| `float` | `pyarrow.float64` |
|
|
| `bool` | `pyarrow.bool` |
|
|
| `str` | `pyarrow.utf8()` |
|
|
| `list` | `pyarrow.List` |
|
|
| `BaseModel` | `pyarrow.Struct` |
|
|
| `Vector(n)` | `pyarrow.FixedSizeList(float32, n)` |
|
|
|
|
LanceDB supports to create Apache Arrow Schema from a
|
|
`pydantic.BaseModel`
|
|
via `lancedb.pydantic.pydantic_to_schema` method.
|
|
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyFrameworksPydanticTypeConversion}
|
|
</CodeBlock>
|
|
|
|
This example shows a more complex Pydantic model with various field types and demonstrates how LanceDB handles:
|
|
- Basic types: `int` and `str` fields
|
|
- Vector fields: `Vector(1536)` creates a fixed-size list of 1536 float32 values
|
|
- List fields: `List[int]` becomes a variable-length list of int64 values
|
|
- Schema generation: The `pydantic_to_schema()` function automatically converts all these types to their Arrow equivalents |