When `Vector` or `MultiVector` declares an integer Arrow value type,
Pydantic currently validates every element as a float. This silently
rounds integers above `2**53`, and converts `uint64`'s maximum value
into an out-of-range float before table insertion.
Use Pydantic's integer schema for integer Arrow value types in both
validators. Floating-point types retain the existing validation. Tests
exercise model serialization and real local table creation, insertion,
and reads for signed and unsigned integer boundaries and floating-point
controls, plus dimensions and optional/nullability behavior. Fractional
floats are rejected for integer declarations; integral floats remain
accepted. This does not extend ANN index support.
Validation on macOS arm64, with the current main native extension built
using Rust 1.97.0:
- Python 3.11.14 and 3.13.11, Pydantic 2.13.5: full `pytest -m "not slow
and not s3_test" -q python/tests` suite: **1,537 passed, 37 skipped, 68
deselected** on each runtime. Comparable unchanged-main baselines:
**1,521 passed, 37 skipped, 68 deselected** each.
- Added cases on unchanged main: **8 failed, 8 passed**; complete
`test_pydantic.py` after the fix: **40 passed** on each runtime.
- Documentation tests with the existing embeddings dependencies in an
isolated environment, Python 3.11.14: **77 passed, 6 skipped**, matching
the unchanged-main baseline. The workflow's Linux/Python 3.13 doctest
matrix was not reproduced locally.
- Repository Ruff lint/format, Pyright, all applicable pre-commit hooks,
compile checks, and `git diff --check` passed. Eight public SDK table
round trips passed on each final installed wheel runtime.
Slow, credential-dependent S3 integration tests, the remaining
OS/runtime matrix, and the minimum Pydantic/PyArrow dependency matrix
were not run locally. The native extension was built from unchanged
current main; the final local wheel repack contains the verified Python
change with that identical native extension. A separate candidate
maturin packaging attempt triggered Rust recompilation and was stopped,
so an official candidate distribution build was not completed. The
project requires Pydantic `>=2.7.4,<3`; Pydantic v1 support was removed
in #3990.
AI assistance: developed and verified with **gpt-6.1-sol**.
Since #3232, `pydantic_to_schema` maps an `Enum` field to the Arrow type
of its values, including enums without a `str`/`int` mixin. Writing such
models still failed: `model_to_dict()` returns `model.model_dump()`,
which keeps the Enum members, and Arrow can only convert members that
are also `str`/`int` instances.
```python
class Color(Enum):
RED = "red"
class Item(LanceModel):
color: Color
tbl = db.create_table("items", schema=Item)
tbl.add([Item(color=Color.RED)])
# pyarrow.lib.ArrowTypeError: Expected bytes, got a 'Color' object
```
`create_table(data=[...])` fails the same way, and int-valued enums
raise `ArrowInvalid`.
`model_to_dict()` now replaces Enum members with their values, including
in nested models and lists. Lists of plain scalars (vectors in
particular) are returned without being walked, so the per-row cost stays
about the same as `model_dump()` alone (21.3µs vs 20.3µs for a 1536-dim
model). Reading back is unchanged: Pydantic already validates the stored
value into the Enum member.
Added `test_plain_enum_values_are_written`, which covers str and int
valued enums, `Optional` and `List` fields, and an Enum inside a nested
model, and checks that `to_pydantic` round-trips the rows. It fails with
the `ArrowTypeError` above before the change. `test_pydantic.py`,
`test_embeddings.py`, `test_table.py` and `test_db.py` pass, and `ruff
format --check` / `ruff check` are clean.
## Summary
- infer fixed-size float32 Arrow vectors from `VectorField` metadata
when a model uses the standard `list[float]` annotation
- document the mypy-compatible embedding-field declaration while
preserving the existing `Vector(dim)` runtime API
- add regression coverage for schema conversion and model construction
## Root cause
`Vector(dim)` is a runtime constrained-type factory, but Python's static
typing rules do not permit call expressions in annotation positions.
Mypy's suggested `Vector[...]` spelling also cannot represent a
dimension computed at runtime and fails because `Vector` is a function.
`VectorField` already carries the embedding function and its dimension,
but schema conversion previously ignored that metadata for a statically
valid `list[float]` annotation.
The runtime factory remains available for dimension-constrained Pydantic
validation. Embedding-backed models that require mypy compatibility can
now use `list[float] = embedding.VectorField()`; LanceDB derives the
same nullable fixed-size float32 Arrow type from the field metadata.
## Validation
- `cd python && uv run --extra tests pytest
python/tests/test_pydantic.py -q` (23 passed)
- focused mypy check reproducing the original error and accepting the
new `LanceModel` declaration
- isolated schema checks with Pydantic 1.10.24 and 2.11.7
- `ruff format .`
- `ruff check .`
Fixes#1982
<!-- lance-gatekeeper-fix:v1 agent=877facbc659cdc9424bd367e12426366
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
LanceDB's Python SDK now requires Pydantic `>=2.7.4,<3` and uses the v2
APIs throughout. This removes dual-version behavior from schema
conversion, query serialization, embedding models, and Function wire
models while preserving their existing public and canonical-wire
behavior.
The minimum-dependencies CI job pins Pydantic 2.7.4 so the declared
compatibility floor remains covered.
## Summary
- raise a clear `TypeError` when `Vector` is used without a dimension
- preserve normal `Vector(dim)` behavior across Pydantic v1 and v2
- add a regression test that defines a model without importing PyArrow
## Root cause
Pydantic interpreted the bare `Vector` factory as a callable field type
and inspected its postponed annotations in the user model's namespace.
Because that namespace did not define LanceDB's internal `pa` alias,
model construction failed with the misleading `NameError: name 'pa' is
not defined` instead of explaining that `Vector` must be parameterized.
The factory now exposes Pydantic's v1 and v2 schema hooks and rejects
bare use before signature introspection with guidance to use
`Vector(dim)`.
## Validation
- `uvx --from 'ruff==0.15.20' ruff check .`
- `uvx --from 'ruff==0.15.20' ruff format --check
python/python/lancedb/pydantic.py python/python/tests/test_pydantic.py`
- `cd python && uv run --extra tests pytest
python/tests/test_pydantic.py::test_bare_vector_raises_clear_error -q`
- `cd python && uv run --extra tests pytest
python/tests/test_pydantic.py -q`
- compatibility checks with Pydantic 1.10.22, 2.11.4, and 2.13.4
Fixes#2384
<!-- lance-gatekeeper-fix:v1 agent=71e7473e18c91db5137a3c0d3bb73640
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Closes#3502
## Problem
A bare, unparameterised `typing.List` / `typing.Tuple` field crashes
`to_arrow_schema` with an opaque `AttributeError: __args__`:
```python
from typing import Tuple
from lancedb.pydantic import LanceModel
class Doc(LanceModel):
items: Tuple
Doc.to_arrow_schema() # AttributeError: __args__
```
In `_py_type_to_arrow_type`, the branch `elif getattr(py_type,
"__origin__", None) in (list, tuple)` is taken for a bare generic (its
`__origin__` is `list / tuple`), but the next line reads
`py_type.__args__[0]`, and a bare generic has no `__args__`. Other
unsupported types (e.g. `Dict[str, int]`) correctly raise a clear
`TypeError`, so this case is inconsistent.
Fix
Guard the element-type lookup with `getattr(py_type, "__args__", None)`
and raise a clear `TypeError` when it is missing, matching the existing
behavior for other unsupported types. Bare builtin list / tuple are
unaffected (their `__origin__` is `None`, so they already fall through
to the existing `TypeError`).
Testing
- Added `test_bare_generic_raises_type_error` covering both `List` and
`Tuple`.
- ruff format and ruff check clean.
## Summary
Fixes#1846.
Python `Enum` fields raised `TypeError: Converting Pydantic type to
Arrow Type: unsupported type <enum 'SomethingTypes'>` when converting a
Pydantic model to an Arrow schema.
The fix adds Enum detection in `_pydantic_type_to_arrow_type`. When an
Enum subclass is encountered, the value type of its members is inspected
and mapped to the appropriate Arrow type:
- `str`-valued enums (e.g. `class Status(str, Enum)`) → `pa.utf8()`
- `int`-valued enums (e.g. `class Priority(int, Enum)`) → `pa.int64()`
- Other homogeneous value types → the Arrow type for that Python type
- Mixed-value or empty enums → `pa.utf8()` (safe fallback)
This covers the common `(str, Enum)` and `(int, Enum)` mixin patterns
used in practice.
## Changes
- `python/python/lancedb/pydantic.py`: add Enum branch in
`_pydantic_type_to_arrow_type`
- `python/python/tests/test_pydantic.py`: add `test_enum_types` covering
`str`, `int`, and `Optional` Enum fields
## Note on #2395
PR #2395 handles `StrEnum` (Python 3.11+) specifically, using a
dictionary-encoded type. This PR handles the broader `(str, Enum)` /
`(int, Enum)` mixin pattern that works across all Python versions and
stores values as their natural Arrow type.
AI assistance was used in developing this fix.
This request improves support for `pydantic` integration by adding
`to_pydantic` method to asynchronous queries and handling models that
use `alias` in field definitions. Fixes#2436 and closes#2437 .
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Added support for converting asynchronous query results to Pydantic
models.
- **Bug Fixes**
- Simplified conversion of query results to Pydantic models for improved
reliability.
- Improved handling of field aliases and computed fields when mapping
query results to Pydantic models.
- **Tests**
- Added tests to verify correct mapping of aliased and computed fields
in both synchronous and asynchronous scenarios.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This PR adds ColPali support with ColPaliEmbeddings class (tagged
"colpali") using ColQwen2.5 for multi-vector text/image embeddings. Also
added MultiVector Pydantic type to handle the vector lists.
I've added some integration test for the embedding model and some unit
test for the new Pydantic type. Could be a template for other ColPali
variants as well. or until transformers🤗 starts supporting it.
Still `TODO`:
- [ ] Documentation
- [ ] Add an example
_Could also allow Image as query, but didn't work well when testing it._
[ColPali-Engine](https://github.com/illuin-tech/colpali) version:
0.3.9.dev17+g3faee24
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced support for ColPali-based multimodal multi-vector
embeddings for both text and images.
- Added a new embedding class for generating multi-vector embeddings,
configurable for various model and processing options.
- Added a new Pydantic type for multi-vector embeddings, supporting
validation and schema generation for lists of fixed-dimension vectors.
- **Bug Fixes**
- Ensured proper asynchronous index creation in query tests for improved
reliability.
- **Tests**
- Added integration tests for ColPali embeddings, including
text-to-image search and validation of multi-vector fields.
- Added comprehensive tests for the new multi-vector Pydantic type,
covering schema, validation, and default value behavior.
- **Chores**
- Updated optional dependencies to include the ColPali engine.
- Added utility to check for availability of flash attention support.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This changes `lancedb` from a "pure python" setuptools project to a
maturin project and adds a rust lancedb dependency.
The async python client is extremely minimal (only `connect` and
`Connection.table_names` are supported). The purpose of this PR is to
get the infrastructure in place for building out the rest of the async
client.
Although this is not technically a breaking change (no APIs are
changing) it is still a considerable change in the way the wheels are
built because they now include the native shared library.
LeonSGP
Mohammad Hijjawi
lancedb-gatefixer[bot]
Xuanwo
Armaan Sandhu
Zelys
Lei Xu
Mykola Skrynnyk
Magnus
Philip MeierandWill Jones
Weston Pace