jasonz-lanceandClaude Sonnet 5
40ff1d52a6
_open_pinned_table() reopened a private handle via _table_from_pickle_state(_table_to_pickle_state(self._table)) so checking it out to the block plan's pinned version would never mutate self._table. That held for local (on-disk) tables, whose pickle state is just reopen coordinates, but not for the other two table kinds: - RemoteTable: the pickle-state helpers special-case remote by returning the *same* object (correct for their original purpose -- cross-process pickling, where the unpickled copy necessarily lives in a different process -- but wrong for same-process reuse here). checkout() is a real in-place mutation on RemoteTable, so every 2-phase read pinned (and, since Cloud rejects writes through a version-pinned handle, effectively made read-only) the caller's own live table object. - memory://: the pickle state *is* the row data (to_arrow()), captured fresh on every _open_pinned_table() call instead of once at construction, so a write made after construction leaked into every later read -- the opposite of what a version pin is supposed to guarantee. A table already past version 1 also failed construction outright, since the rebuilt table is always a fresh version-1 table. Fix: deep-copy the RemoteTable instead (round-trips through its own __getstate__/__setstate__, which carries only reopen coordinates and clears the live connection handle, so the copy reconnects independently on first use). For local/memory tables, capture _pinned_table_state once at construction instead of recomputing it per call, and skip checkout() for memory tables -- a reopened memory table already is exactly the pinned snapshot by construction, and checking out a version number on it would fail since it only has one. Surfaced in the process: a reopened memory table's to_lance() always failed (pylance's raw lance.dataset() open can't resolve an isolated in-memory store), which forced every memory-table block read through the slower generic query builder with no error -- previously a gatekeeper-acknowledged non-blocking limitation, since it failed loudly rather than returning wrong data. _pinned_lance_dataset now recognizes a memory:// table up front (not only RemoteTable's NotImplementedError) and skips the native fast path for it, the same way it already does for RemoteTable, so memory-table block reads now also work correctly instead of merely failing safely. Added 3 regression tests: the caller's RemoteTable handle must stay unpinned after iterating a 2-phase dataset over a mocked remote table; a memory table's pin must survive a write made after construction; and constructing over a memory table already past version 1 must not raise. Full suite: 210 passed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
LanceDB Python SDK
A Python library for LanceDB.
Installation
pip install lancedb
Pre-Haswell x86_64 hosts: lancedb-compat
The default lancedb wheel targets x86-64-haswell (AVX2 + FMA + F16C) for full performance on modern hardware. Pre-Haswell hosts — Intel Sandy Bridge / Ivy Bridge / Westmere; AMD Bulldozer / Piledriver / Steamroller — don't have AVX2 and crash with Illegal instruction at import lancedb.
For those hosts, install the lancedb-compat package instead:
pip install lancedb-compat
Same Python API (import lancedb works as usual). The compat wheel is compiled at the x86-64-v2 baseline (Nehalem-class) and uses runtime SIMD dispatch in the embedded lance crate to pick the right kernel tier (scalar / AVX / AVX+FMA / AVX2+FMA / AVX-512) at load time, so it still goes fast on modern hardware while running cleanly on the pre-Haswell silicon. Use lance.simd_info() from Python to verify which tier was selected.
lancedb and lancedb-compat install to the same lancedb/ namespace and conflict at install time. Pick one. To switch, pip uninstall lancedb first, then pip install lancedb-compat (or vice-versa).
If you need a custom baseline (or lancedb-compat isn't yet published for your platform), build from source with the override:
RUSTFLAGS="-C target-cpu=x86-64-v2" maturin build --release
pip install ./target/wheels/lancedb-*.whl
Preview Releases
Stable releases are created about every 2 weeks. For the latest features and bug fixes, you can install the preview release. These releases receive the same level of testing as stable releases, but are not guaranteed to be available for more than 6 months after they are released. Once your application is stable, we recommend switching to stable releases.
pip install --pre --extra-index-url https://pypi.fury.io/lancedb/ lancedb
Threading in CPU-limited containers
LanceDB uses separate pools for compute work and storage I/O. On a container with two visible CPUs, current releases intentionally use one compute worker by default; no manual configuration is needed. If every query logs an I/O core reservation warning on a two-CPU container, upgrade from LanceDB 0.21.1 or earlier.
The two commonly tuned environment variables control different resources:
LANCE_CPU_THREADSoverrides the number of compute workers. One worker is the appropriate setting for a two-CPU container when an explicit override is needed.LANCE_IO_THREADScontrols concurrent storage operations, not reserved CPU cores. Its default can be greater than the number of CPUs because I/O workers spend much of their time waiting for storage.
Keep the defaults unless measurements show that the workload benefits from an override. See the Lance threading model for the current defaults and tuning guidance.
Usage
Basic Example
import lancedb
db = lancedb.connect('<PATH_TO_LANCEDB_DATASET>')
table = db.open_table('my_table')
results = table.search([0.1, 0.3]).limit(20).to_list()
print(results)
Development
See CONTRIBUTING.md for information on how to contribute to LanceDB.