rust/lancedb
1
0
Fork 0
mirror of https://github.com/lancedb/lancedb.git synced 2026-10-25 13:12:39 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(python): align pylance extra and preview install indexes (#4556)

Browse Source 
Fixes #4248.

The preview index still contains pylance 1.0.0b6. uv's default
first-index resolution can select it, or backtrack to an old LanceDB
release when installing the extra. Meanwhile, the extra still permits
pylance 5 even though current main bundles Lance 14.

Raise the optional dependency to `pylance>=14.0.0b12`, the version
already pinned in the test extra, and keep the lockfile metadata
consistent. Document both Fury indexes and uv's `unsafe-best-match`
strategy in the Python README and the source-owned quickstart
(`docs/web/quickstart.mdx`). Explain the index trust requirement and
recommend installing the extra to enforce the version floor.

Add CI checks that reject older Lance majors and ensure the pylance and
tests extras remain compatible. Run these checks when either dependency
manifest changes, so a future Lance major bump cannot silently leave the
Python minimum behind.

### Validation

- Recreated the reported install on Ubuntu 26.04 / CPython 3.12.15 with
uv 0.12.23: `lancedb==0.40.0b5` plus the old index selects
`pylance==1.0.0b6`. Reproduced both the missing `blob_array` import and
the unexpected `blob_handling` scanner argument using a real blob table.
- Resolved the modified `python/pyproject.toml` with the pylance extra
against both indexes: selects `pylance==14.0.0b18`, matching the current
Rust pin.
- Ran both documented pip and uv installations in isolated environments:
`lancedb==0.41.0b7`, `pylance==14.0.0b18`; blob-array construction and a
blob-bytes round trip through `to_pandas(blob_mode="bytes")` pass. Both
package dependency checks pass.
- New regression fails on the original dependency bound (27
rejected-version cases incorrectly accepted), then passes with the fix.
Full CI-script suite: 3 tests pass.
- `uv lock --check` with both indexes passes; focused Ruff checks pass;
`git diff --check` passes.
- Repository-wide Ruff reports 49 existing errors in untouched
`docs/web-tests` files. Unrelated formatting changes were discarded.

Runtime smoke checks used released native wheels; source dependency
resolution was tested directly from the modified pyproject. No Rust or
Python runtime code changes.

---------

Co-authored-by: Xuanwo <github@xuanwo.io>
This commit is contained in:
KarthikS
2026-10-24 03:27:32 +08:00
committed by GitHub
co-authored by Xuanwo
parent 5a79c7d22d
commit 702389654e
7 changed files with 146 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.agents/skills/lancedb-update-lance-dependency/SKILL.md
+27
View File
@@ -48,6 +48,23 @@ Inputs can be a version (`7.2.0-beta.1`), a tag (`v7.2.0-beta.1`), a tag ref (`r
This updates the Rust workspace Lance dependencies through `ci/set_lance_version.py`, updates `java/pom.xml`, refreshes Cargo metadata, and prints JSON metadata containing `branch_name`, `commit_message`, and `pr_title`.
Also synchronize the Python requirements in `python/pyproject.toml`: set the
`tests` extra's exact `pylance` pin to the matching published Python release,
and update the `pylance` extra's minimum when needed to exclude older bundled
Lance majors. The minimum must still accept the test pin. Python prerelease
spelling differs from Rust (for example, `15.0.0-beta.1` becomes `15.0.0b1`).
Confirm the matching Python release is published before changing the pin.
Refresh and check `python/uv.lock` from the repository root:
```bash
uv lock --project python --index https://pypi.fury.io/lancedb/ --index https://pypi.fury.io/lance-format/ --index-strategy unsafe-best-match
uv lock --project python --check --index https://pypi.fury.io/lancedb/ --index https://pypi.fury.io/lance-format/ --index-strategy unsafe-best-match
```
This strategy considers versions across indexes; use it only with trusted
indexes. Commit the Python requirements and lockfile together with the bump.
5. Run validation:
```bash
@@ -57,6 +74,16 @@ Inputs can be a version (`7.2.0-beta.1`), a tag (`v7.2.0-beta.1`), a tag ref (`r
Fix real diagnostics and rerun clippy until it succeeds. Do not skip warnings.
Run the Python dependency guards with Python 3.11+ and `packaging` installed:
```bash
python3 -m unittest discover -s ci/tests -p test_python_dependencies.py -v
```
These checks reject older Lance majors and ensure the `pylance` and `tests`
extras remain compatible. The full CI-script suite uses
`python3 -m unittest discover -s ci/tests -v`.
6. Inspect `git status --short` and `git diff` to ensure only the Lance dependency update and required compatibility fixes are present.
7. If the task only asks to prepare local changes, stop here and report the changed files and validation result.
.github/workflows/ci-scripts.yml
+6
View File
@@ -5,6 +5,8 @@ on:
branches:
- main
paths:
- Cargo.toml
- python/pyproject.toml
- ci/check_cargo_lock.py
- ci/check_sdist.py
- ci/set_lance_version.py
@@ -12,6 +14,8 @@ on:
- .github/workflows/ci-scripts.yml
pull_request:
paths:
- Cargo.toml
- python/pyproject.toml
- ci/check_cargo_lock.py
- ci/check_sdist.py
- ci/set_lance_version.py
@@ -30,5 +34,7 @@ jobs:
- uses: actions/setup-python@v6
with:
python-version: "3.13"
- name: Install test dependencies
run: python -m pip install packaging
- name: Run tests
run: python -m unittest discover -s ci/tests -v
ci/tests/test_python_dependencies.py
+47
View File
@@ -0,0 +1,47 @@
import tomllib
import unittest
from pathlib import Path
from packaging.requirements import Requirement
from packaging.version import Version
REPO_ROOT = Path(__file__).resolve().parents[2]
class PythonDependenciesTest(unittest.TestCase):
def setUp(self):
with (REPO_ROOT / "python/pyproject.toml").open("rb") as f:
self.project = tomllib.load(f)["project"]
self.pylance = Requirement(self.project["optional-dependencies"]["pylance"][0])
def test_pylance_extra_excludes_older_lance_majors(self):
with (REPO_ROOT / "Cargo.toml").open("rb") as f:
lance = tomllib.load(f)["workspace"]["dependencies"]["lance"]
if "version" not in lance:
self.skipTest("Local Lance checkout has no version requirement")
major = Version(lance["version"].lstrip("=")).major
for older_major in range(major):
for suffix in (".0.0", ".0.0b6", ".999.999"):
version = f"{older_major}{suffix}"
with self.subTest(version=version):
self.assertFalse(
self.pylance.specifier.contains(version, prereleases=True),
f"The pylance extra must exclude Lance {version}; "
f"the bundled Lance is major {major}",
)
def test_pylance_extra_accepts_tested_version(self):
requirements = [
Requirement(dep) for dep in self.project["optional-dependencies"]["tests"]
]
tested = next(dep for dep in requirements if dep.name == "pylance")
pin = next(spec for spec in tested.specifier if spec.operator == "==")
self.assertTrue(
self.pylance.specifier.contains(pin.version, prereleases=True),
f"The tests and pylance extras must be installable together: {tested}",
)
if __name__ == "__main__":
unittest.main()
docs/web/quickstart.mdx
+27
View File
@@ -58,6 +58,33 @@ uv add --prerelease allow --index https://pypi.fury.io/lancedb/ lancedb
```
</CodeGroup>
If you also need the optional `lance` Python API, install the `pylance` extra and
include the Lance preview index:
<CodeGroup>
```bash Python (pip) icon="terminal"
pip install --pre \
--extra-index-url https://pypi.fury.io/lancedb/ \
--extra-index-url https://pypi.fury.io/lance-format/ \
"lancedb[pylance]"
```
```bash Python (uv) icon="terminal"
uv pip install --prerelease allow \
--index https://pypi.fury.io/lancedb/ \
--index https://pypi.fury.io/lance-format/ \
--index-strategy unsafe-best-match \
"lancedb[pylance]"
```
</CodeGroup>
The LanceDB preview index contains old `pylance` releases; current Lance previews
are published to the `lance-format` index. By default, uv only considers versions
from the first index containing a package, which can select an old `pylance` or
backtrack to an older LanceDB release. `unsafe-best-match` searches all configured
indexes, like pip; use it only with indexes you trust. Install the extra rather
than an unconstrained `pylance` so LanceDB's minimum supported version is enforced.
<Note>
Pre-release builds receive the same level of testing as stable releases, but their availability is not guaranteed
for more than 6 months after release. For real-world workloads, we recommend you use the latest stable release
python/README.md
+27
View File
@@ -40,6 +40,33 @@ Stable releases are created about every 2 weeks. For the latest features and bug
pip install --pre --extra-index-url https://pypi.fury.io/lancedb/ lancedb
```
If you also need the optional `lance` Python API, install the `pylance` extra
and include the Lance preview index:
```bash
pip install --pre \
--extra-index-url https://pypi.fury.io/lancedb/ \
--extra-index-url https://pypi.fury.io/lance-format/ \
"lancedb[pylance]"
```
With uv, also select the best matching version across the indexes:
```bash
uv pip install --prerelease allow \
--index https://pypi.fury.io/lancedb/ \
--index https://pypi.fury.io/lance-format/ \
--index-strategy unsafe-best-match \
"lancedb[pylance]"
```
The LanceDB preview index contains old `pylance` releases; current Lance previews
are published to the `lance-format` index. By default, uv only considers versions
from the first index containing a package, which can select an old `pylance` or
backtrack to an older LanceDB release. `unsafe-best-match` searches all configured
indexes, like pip; use it only with indexes you trust. Install the extra rather
than an unconstrained `pylance` so LanceDB's minimum supported version is enforced.
### Threading in CPU-limited containers
LanceDB uses separate pools for compute work and storage I/O. On a container with
python/pyproject.toml
+3 -2
View File
@@ -45,7 +45,8 @@ repository = "https://github.com/lancedb/lancedb"
[project.optional-dependencies]
pylance = [
"pylance>=5.0.0b5",
# Keep the minimum in the bundled Lance major and compatible with tests below.
"pylance>=15.0.0b1",
]
# A library only needs the OpenTelemetry API; the application supplies and
# configures the SDK (the actual exporter/reader). See
@@ -63,7 +64,7 @@ tests = [
"polars>=0.19, <=1.32.3",
"pyarrow<25",
"pyarrow-stubs>=16.0",
"pylance==14.0.0b12",
"pylance==15.0.0b1",
"requests>=2.31.0",
"datafusion>=54,<55",
"opentelemetry-sdk>=1.30.0",
python/uv.lock
Generated
+9 -9
View File
@@ -2006,8 +2006,8 @@ requires-dist = [
{ name = "pyarrow", marker = "extra == 'tests'", specifier = "<25" },
{ name = "pyarrow-stubs", marker = "extra == 'tests'", specifier = ">=16.0" },
{ name = "pydantic", specifier = ">=2.7.4,<3" },
{ name = "pylance", marker = "extra == 'pylance'", specifier = ">=5.0.0b5" },
{ name = "pylance", marker = "extra == 'tests'", specifier = "==14.0.0b12" },
{ name = "pylance", marker = "extra == 'pylance'", specifier = ">=15.0.0b1" },
{ name = "pylance", marker = "extra == 'tests'", specifier = "==15.0.0b1" },
{ name = "pyright", marker = "extra == 'dev'", specifier = ">=1.1.350" },
{ name = "pytest", marker = "extra == 'tests'", specifier = ">=7.0" },
{ name = "pytest-asyncio", marker = "extra == 'tests'", specifier = ">=0.21" },
@@ -3938,7 +3938,7 @@ crypto = [
[[package]]
name = "pylance"
version = "14.0.0b12"
version = "15.0.0b1"
source = { registry = "https://pypi.fury.io/lance-format/" }
dependencies = [
{ name = "lance-namespace" },
@@ -3948,12 +3948,12 @@ dependencies = [
{ name = "pyarrow" },
]
wheels = [
{ url = "https://pypi.fury.io/lance-format/-/ver_1Ixasc/pylance-14.0.0b12-cp310-abi3-macosx_11_0_arm64.whl", hash = "sha256:84350a3c5c1eea3398f4c4fad7cfe85b831d831a5e43b5ca2a783dbdb0e11969" },
{ url = "https://pypi.fury.io/lance-format/-/ver_sCEkG/pylance-14.0.0b12-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:76b386c182a6a8e7b86fbef28f84d1dd39e4626ac8681a0cf36f40a45dd7cb87" },
{ url = "https://pypi.fury.io/lance-format/-/ver_1vdT1C/pylance-14.0.0b12-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:6aa548c7937fcdfa8f1943a2d59041a42f8b500199880dbb509c7195fa253381" },
{ url = "https://pypi.fury.io/lance-format/-/ver_2fXu1V/pylance-14.0.0b12-cp310-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:79f82d6a6c50f37aab338470e3733436e6a27fbb5972d29fcf453ece56d3b3c6" },
{ url = "https://pypi.fury.io/lance-format/-/ver_24UavN/pylance-14.0.0b12-cp310-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:de0305999257002785f60169e6926896584d66f1da050bb5afd1d48c6e3d9bb9" },
{ url = "https://pypi.fury.io/lance-format/-/ver_XNynX/pylance-14.0.0b12-cp310-abi3-win_amd64.whl", hash = "sha256:4d0cdb689e898ce71e8e434d00a3b97639cea8fb97da21f1c921543be363268f" },
{ url = "https://pypi.fury.io/lance-format/-/ver_23L1Nd/pylance-15.0.0b1-cp310-abi3-macosx_11_0_arm64.whl", hash = "sha256:19605d3daca1224cc0db079054be5d4eec403dcf286bf4617430bfc259ab0a52" },
{ url = "https://pypi.fury.io/lance-format/-/ver_1Ho3RK/pylance-15.0.0b1-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:be22b0b6f1e1d250ffaa82b449e982c817db33103c483d1005020285310623cf" },
{ url = "https://pypi.fury.io/lance-format/-/ver_1QmyF0/pylance-15.0.0b1-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:a5b2761d60b1b9db112058213f1f7b2de82b08cbe65888fab08b0a56e702482e" },
{ url = "https://pypi.fury.io/lance-format/-/ver_1in2Ph/pylance-15.0.0b1-cp310-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:294a0fedae312ed32d1b66cddbf7e69770ec9b4c82a873af60c619ed88450718" },
{ url = "https://pypi.fury.io/lance-format/-/ver_21xpLH/pylance-15.0.0b1-cp310-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:ff20a16703d4d3f3bc675b25ce1871ba1d54137929dc46305bffdd07f7b35a6b" },
{ url = "https://pypi.fury.io/lance-format/-/ver_AAGzF/pylance-15.0.0b1-cp310-abi3-win_amd64.whl", hash = "sha256:3b134e24a5becfc940ca134cf78aa3bae8b80ec40f8927a1b856273ab76213f2" },
]
[[package]]