Daniel RammerandClaude Opus 5
106f9ddb31
`checkpoint_lsm` looped flush → compact and read `generations_remaining` out of each compact response. Two problems. That count is stale the moment a concurrent write lands, so on a table taking writes "L0 is empty" never becomes true and the loop only ever exited on its deadline. And carrying it back at all meant eight public types describing work already visible in `get_lsm_stats`. It now fixes a target watermark — the newest generation per bucket, read after the opening flush — then triggers compaction and polls until no generation at or below that target remains. Generations created *while* it runs are deliberately not waited on, which is what makes it terminate under sustained writes and is why the deadline could go. * Deleted: `CheckpointOptions`, `CheckpointReport`, `CheckpointStopReason`, `CompactOptions`, `CompactReport`, `CompactBucketReport`, `FlushReport`, `FlushBucketReport`. `checkpoint_lsm`, `flush_lsm` and `compact_lsm` all return `()`; observability is `get_lsm_stats` before and after. * Progress, not time, is the bound: `MAX_IDLE_POLLS` consecutive polls with neither a drop in outstanding generations nor a running pass is an error. A saturated pool or a fenced writer fails loudly instead of spinning; a slow table waits, since a running pass resets the counter. Wrap the call in `tokio::time::timeout` for a wall-clock bound. * The metric counts generations, not buckets. One pass drains a bounded prefix, so a per-bucket boolean reads as "no progress" for every pass but the last — a bucket needing more passes than the bound would fail a checkpoint that was working. * `BucketStats` gains `compacting`, and the loop never triggers against a bucket already compacting: it would only collect 429s from the latch doing the work. * `get_lsm_stats` takes `include_generation_rows` again, off by default. Each count opens an uncached Lance dataset and this loop polls the route. * `send_lsm_route` splits from its JSON decode — flush and compact answer 202 with no body, and decoding an empty string is a parse error. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
LanceDB Python SDK
A Python library for LanceDB.
Installation
pip install lancedb
Pre-Haswell x86_64 hosts: lancedb-compat
The default lancedb wheel targets x86-64-haswell (AVX2 + FMA + F16C) for full performance on modern hardware. Pre-Haswell hosts — Intel Sandy Bridge / Ivy Bridge / Westmere; AMD Bulldozer / Piledriver / Steamroller — don't have AVX2 and crash with Illegal instruction at import lancedb.
For those hosts, install the lancedb-compat package instead:
pip install lancedb-compat
Same Python API (import lancedb works as usual). The compat wheel is compiled at the x86-64-v2 baseline (Nehalem-class) and uses runtime SIMD dispatch in the embedded lance crate to pick the right kernel tier (scalar / AVX / AVX+FMA / AVX2+FMA / AVX-512) at load time, so it still goes fast on modern hardware while running cleanly on the pre-Haswell silicon. Use lance.simd_info() from Python to verify which tier was selected.
lancedb and lancedb-compat install to the same lancedb/ namespace and conflict at install time. Pick one. To switch, pip uninstall lancedb first, then pip install lancedb-compat (or vice-versa).
If you need a custom baseline (or lancedb-compat isn't yet published for your platform), build from source with the override:
RUSTFLAGS="-C target-cpu=x86-64-v2" maturin build --release
pip install ./target/wheels/lancedb-*.whl
Preview Releases
Stable releases are created about every 2 weeks. For the latest features and bug fixes, you can install the preview release. These releases receive the same level of testing as stable releases, but are not guaranteed to be available for more than 6 months after they are released. Once your application is stable, we recommend switching to stable releases.
pip install --pre --extra-index-url https://pypi.fury.io/lancedb/ lancedb
Usage
Basic Example
import lancedb
db = lancedb.connect('<PATH_TO_LANCEDB_DATASET>')
table = db.open_table('my_table')
results = table.search([0.1, 0.3]).limit(20).to_list()
print(results)
Development
See CONTRIBUTING.md for information on how to contribute to LanceDB.