XY Zhan
723a2394e5
**The maintained set is an intent, not a snapshot.** An unnamed set is stored as the intent to maintain every index, which lance re-resolves whenever it builds a MemTable, so an index created after the spec installs is maintained too. A named set is checked here, where the caller can be told which names exist; an index of a kind the MemWAL cannot mirror fails a named set and is skipped by an unnamed one. **A newly created index reaches a writer that is already open.** lancedb caches a `ShardWriter` per shard and reuses it, so `create_index` installs the current set on the cached writer. A failure there is reported rather than propagated: the index is committed durably by then, and the writer this process holds may be fenced or poisoned, which is no reason to fail creating an index. **A table carries one spec.** `set_lsm_write_spec` fails while one is installed and names `unset_lsm_write_spec` in the error, because the generations already written were homed under that spec. Accepting a repeat that changed only the maintained set would hand a caller who also passed a different sharding a success that changed nothing they asked for. **The read path asks what is actually carried.** An unnamed set leaves the stored list empty, so the list alone cannot decide whether an index is maintained. The read asks the spec *and* every resident MemTable whether it carries the index, so a MemTable built before the index joined the set refuses the read rather than answering without its rows. Both spec routes report an unnamed set as `null` rather than `[]`: maintain whatever the table gains, and maintain nothing, are different settings. ## Known gap The guard above covers resident MemTables. Once a writer flushes and closes none is resident, so a retained generation written before the index existed still gets an indexed read that omits its rows. This window is **wider than on main**: a set resolved at install time excluded an index created later, so the read was refused; an intent includes it, and the refusal only holds while a MemTable is resident. Closing it needs the generation's own coverage, which this layer cannot see — its snapshot carries a generation and a path — so it belongs in lance's scanner or flush path. ## Testing `cargo nextest run -p lancedb --lib --features remote` — **1479 passed, 3 skipped**. `cargo fmt --all --check`, and `cargo clippy --features remote --lib --tests --examples -- -D warnings`, both clean. Covered: the round trip of an unnamed set, a set over an installed spec refused, a spec replaced through unset, a read refused while a resident MemTable predates the index, and an index created mid-write reaching the open writer. Two Python binding tests move with the semantics: an unnamed set now reads back as `None` rather than the list it resolves to, and the refusal is matched on `already set`. Neither was run locally — no Python build here — so CI is the check on them.
LanceDB Python SDK
A Python library for LanceDB.
Installation
pip install lancedb
Pre-Haswell x86_64 hosts: lancedb-compat
The default lancedb wheel targets x86-64-haswell (AVX2 + FMA + F16C) for full performance on modern hardware. Pre-Haswell hosts — Intel Sandy Bridge / Ivy Bridge / Westmere; AMD Bulldozer / Piledriver / Steamroller — don't have AVX2 and crash with Illegal instruction at import lancedb.
For those hosts, install the lancedb-compat package instead:
pip install lancedb-compat
Same Python API (import lancedb works as usual). The compat wheel is compiled at the x86-64-v2 baseline (Nehalem-class) and uses runtime SIMD dispatch in the embedded lance crate to pick the right kernel tier (scalar / AVX / AVX+FMA / AVX2+FMA / AVX-512) at load time, so it still goes fast on modern hardware while running cleanly on the pre-Haswell silicon. Use lance.simd_info() from Python to verify which tier was selected.
lancedb and lancedb-compat install to the same lancedb/ namespace and conflict at install time. Pick one. To switch, pip uninstall lancedb first, then pip install lancedb-compat (or vice-versa).
If you need a custom baseline (or lancedb-compat isn't yet published for your platform), build from source with the override:
RUSTFLAGS="-C target-cpu=x86-64-v2" maturin build --release
pip install ./target/wheels/lancedb-*.whl
Preview Releases
Stable releases are created about every 2 weeks. For the latest features and bug fixes, you can install the preview release. These releases receive the same level of testing as stable releases, but are not guaranteed to be available for more than 6 months after they are released. Once your application is stable, we recommend switching to stable releases.
pip install --pre --extra-index-url https://pypi.fury.io/lancedb/ lancedb
Threading in CPU-limited containers
LanceDB uses separate pools for compute work and storage I/O. On a container with two visible CPUs, current releases intentionally use one compute worker by default; no manual configuration is needed. If every query logs an I/O core reservation warning on a two-CPU container, upgrade from LanceDB 0.21.1 or earlier.
The two commonly tuned environment variables control different resources:
LANCE_CPU_THREADSoverrides the number of compute workers. One worker is the appropriate setting for a two-CPU container when an explicit override is needed.LANCE_IO_THREADScontrols concurrent storage operations, not reserved CPU cores. Its default can be greater than the number of CPUs because I/O workers spend much of their time waiting for storage.
Keep the defaults unless measurements show that the workload benefits from an override. See the Lance threading model for the current defaults and tuning guidance.
Usage
Basic Example
import lancedb
db = lancedb.connect('<PATH_TO_LANCEDB_DATASET>')
table = db.open_table('my_table')
results = table.search([0.1, 0.3]).limit(20).to_list()
print(results)
Development
See CONTRIBUTING.md for information on how to contribute to LanceDB.